[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:22 dist-test-slave-6lwt krb5kdc[2783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:22 dist-test-slave-6lwt krb5kdc[2783](info): set up 2 sockets
Sep 24 10:44:22 dist-test-slave-6lwt krb5kdc[2783](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 10:44:23.640841 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:23 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:23.759859 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37833
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:23.863428 2796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:23.863651 2796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:23.863688 2796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:23.867563 2796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:23.867615 2796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:23.867635 2796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:23.867650 2796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:23.867664 2796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:23.872538 2796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:23.873479 2796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:23.874444 2796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:23.882606 2804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:23.882797 2796 server_base.cc:1034] running on GCE node
W20260924 10:44:23.882679 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:23.882711 2801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:23.883477 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:23.884470 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:23.885650 2796 hybrid_clock.cc:648] HybridClock initialized: now 1790246663885600 us; error 64 us; skew 500 ppm
Sep 24 10:44:23 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246663, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:23.892187 2796 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:23.893096 2796 webserver.cc:460] Webserver started at http://127.2.182.62:40723/ using document root <none> and password file <none>
I20260924 10:44:23.893641 2796 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:23.893728 2796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:23.893950 2796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:23.895854 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "8bd931e920154f74b8263599ef2fdeb9"
format_stamp: "Formatted at 2026-09-24 10:44:23 on dist-test-slave-6lwt"
server_key: "d8e06a3d02352616f433fc7ad2a3cfc1"
server_key_iv: "79c113ea536e2295142c0f4818ad40dc"
server_key_version: "encryptionkey@0"
I20260924 10:44:23.896402 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "8bd931e920154f74b8263599ef2fdeb9"
format_stamp: "Formatted at 2026-09-24 10:44:23 on dist-test-slave-6lwt"
server_key: "d8e06a3d02352616f433fc7ad2a3cfc1"
server_key_iv: "79c113ea536e2295142c0f4818ad40dc"
server_key_version: "encryptionkey@0"
I20260924 10:44:23.899991 2796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:23.902297 2811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:23.903442 2796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:23.903584 2796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "8bd931e920154f74b8263599ef2fdeb9"
format_stamp: "Formatted at 2026-09-24 10:44:23 on dist-test-slave-6lwt"
server_key: "d8e06a3d02352616f433fc7ad2a3cfc1"
server_key_iv: "79c113ea536e2295142c0f4818ad40dc"
server_key_version: "encryptionkey@0"
I20260924 10:44:23.903690 2796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:23.922662 2796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:23.925941 2796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:23.926159 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:23.933287 2796 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:37833
I20260924 10:44:23.933313 2863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:37833 every 8 connection(s)
I20260924 10:44:23.934160 2796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:23.935595 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 2796
I20260924 10:44:23.935775 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:23.937392 2864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:23.937418 2776 external_mini_cluster.cc:1428] Setting key f2ca4017281f0c3cde19d650f889e5eb
I20260924 10:44:23.943463 2864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9: Bootstrap starting.
I20260924 10:44:23.945870 2864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:23.946697 2864 log.cc:824] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:23.948689 2864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9: No bootstrap required, opened a new log
I20260924 10:44:23.952785 2864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } }
I20260924 10:44:23.953073 2864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:23.953147 2864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd931e920154f74b8263599ef2fdeb9, State: Initialized, Role: FOLLOWER
I20260924 10:44:23.953586 2864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } }
I20260924 10:44:23.953729 2864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:23.953815 2864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:23.953908 2864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:23.954764 2864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } }
I20260924 10:44:23.955114 2864 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd931e920154f74b8263599ef2fdeb9; no voters:
Sep 24 10:44:23 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:23.955408 2864 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:23.955513 2869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:23.955749 2869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd931e920154f74b8263599ef2fdeb9, State: Running, Role: LEADER
I20260924 10:44:23.956058 2869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } }
I20260924 10:44:23.956632 2864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:23.957795 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd931e920154f74b8263599ef2fdeb9. Latest consensus state: current_term: 1 leader_uuid: "8bd931e920154f74b8263599ef2fdeb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } } }
I20260924 10:44:23.957795 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd931e920154f74b8263599ef2fdeb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd931e920154f74b8263599ef2fdeb9" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37833 } } }
I20260924 10:44:23.957942 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:23.957942 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:23.958338 2878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:23.961301 2878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:23.963428 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:23.940872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.2.182.62:37833)
0924 10:44:23.941333 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:23.941350 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:23.942294 (+ 944us) server_negotiation.cc:407] Connection header received
0924 10:44:23.945436 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:23.945455 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:23.945825 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:23.946200 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:23.948411 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:23.949912 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:23.951108 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:23.951888 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:23.952071 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:23.956171 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:23.956177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:23.956190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:23.956219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:23.958510 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:23.959204 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:23.959213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:23.959221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:23.959392 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:23.959821 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:23.959825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:23.959826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:23.960299 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:23.960491 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:23.962233 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0924 10:44:23.962509 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260924 10:44:23.967127 2878 catalog_manager.cc:1347] Generated new cluster ID: e605878cd95f4d1581fa2d6ef5d8e32e
I20260924 10:44:23.967185 2878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:23.972604 2878 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:23.973742 2878 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:23.982903 2878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd931e920154f74b8263599ef2fdeb9: Generated new TSK 0
I20260924 10:44:23.983577 2878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:24.012035 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:24.115903 2890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:24.116142 2890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:24.116228 2890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:24.120035 2890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:24.120108 2890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:24.120193 2890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:24.125109 2890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:24.126000 2890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:24.126886 2890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:24.135797 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:24.135857 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:24.135890 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:24.135968 2890 server_base.cc:1034] running on GCE node
I20260924 10:44:24.136487 2890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:24.137087 2890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:24.138236 2890 hybrid_clock.cc:648] HybridClock initialized: now 1790246664138219 us; error 30 us; skew 500 ppm
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:24.140707 2890 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:24.141537 2890 webserver.cc:460] Webserver started at http://127.2.182.1:39001/ using document root <none> and password file <none>
I20260924 10:44:24.142045 2890 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:24.142108 2890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:24.142349 2890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:24.144212 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "a905275797f74edab1513c59a1a5b69c"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "142b1b01b3dda288b892d9287fde5c26"
server_key_iv: "0c2c324cc170a1d99c96d0b0dff7aa6e"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.144716 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "a905275797f74edab1513c59a1a5b69c"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "142b1b01b3dda288b892d9287fde5c26"
server_key_iv: "0c2c324cc170a1d99c96d0b0dff7aa6e"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.148113 2890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:44:24.150348 2905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.151527 2890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:24.151611 2890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "a905275797f74edab1513c59a1a5b69c"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "142b1b01b3dda288b892d9287fde5c26"
server_key_iv: "0c2c324cc170a1d99c96d0b0dff7aa6e"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.151682 2890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:24.165351 2890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:24.168262 2890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:24.168377 2890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:24.168787 2890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:24.169473 2890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:24.169514 2890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.169548 2890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:24.169560 2890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.177393 2890 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40979
I20260924 10:44:24.177402 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40979 every 8 connection(s)
I20260924 10:44:24.178303 2890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:24.187489 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 2890
I20260924 10:44:24.187613 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:24.187892 2776 external_mini_cluster.cc:1428] Setting key 3e01312b99f788a292b8f30255f4760c
I20260924 10:44:24.190433 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.179816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:52329 (local address 127.2.182.62:37833)
0924 10:44:24.179939 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:24.179948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:24.180606 (+ 658us) server_negotiation.cc:407] Connection header received
0924 10:44:24.181866 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:24.181869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:24.181913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:24.181982 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:24.182984 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.183517 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.184547 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.184765 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.184835 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.186990 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:24.186993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:24.186995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:24.187016 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:24.188494 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.189012 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.189021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.189023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.189100 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.189477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.189479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.189480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.189636 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:24.189703 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:24.190234 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 10:44:24.190328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:44:24.191311 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.180066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37833 (local address 127.2.182.1:52329)
0924 10:44:24.180456 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:24.180492 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:24.181685 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:24.182130 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:24.182135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:24.182427 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:24.182858 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.182871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.183613 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.183616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.184449 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.184457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.184874 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.184877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.185067 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.185520 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:24.185533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:24.186848 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:24.188615 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.188619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.188628 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.188858 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.189214 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.189216 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.189217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.189378 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.189740 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:24.189747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:24.190015 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 10:44:24.190188 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 10:44:24.190485 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":104,"threads_started":1}
I20260924 10:44:24.192453 3019 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37833
I20260924 10:44:24.192665 3019 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:24.193146 3019 heartbeater.cc:511] Master 127.2.182.62:37833 requested a full tablet report, sending...
I20260924 10:44:24.194463 2828 ts_manager.cc:194] Registered new tserver with Master: a905275797f74edab1513c59a1a5b69c (127.2.182.1:40979)
I20260924 10:44:24.195531 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:52329
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:24.213267 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:24.318145 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:24.318356 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:24.318395 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:24.322603 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:24.322656 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:24.322714 3024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:24.327652 3024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:24.328538 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:24.329583 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:24.338161 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:24.338224 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:24.338450 3024 server_base.cc:1034] running on GCE node
W20260924 10:44:24.338544 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:24.338976 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:24.339540 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:24.340708 3024 hybrid_clock.cc:648] HybridClock initialized: now 1790246664340695 us; error 24 us; skew 500 ppm
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:24.343330 3024 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:24.344031 3024 webserver.cc:460] Webserver started at http://127.2.182.2:37703/ using document root <none> and password file <none>
I20260924 10:44:24.344497 3024 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:24.344550 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:24.344723 3024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:24.346583 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "5a246e6ffa9c41e484b4fb0535318db4"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "2d0643dc4945cedb8f8531f8f9db82c0"
server_key_iv: "0df864bc7d3f7126f6cbf4a67ec4ab03"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.347055 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "5a246e6ffa9c41e484b4fb0535318db4"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "2d0643dc4945cedb8f8531f8f9db82c0"
server_key_iv: "0df864bc7d3f7126f6cbf4a67ec4ab03"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.350478 3024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:24.352579 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.353742 3024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:24.353828 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "5a246e6ffa9c41e484b4fb0535318db4"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "2d0643dc4945cedb8f8531f8f9db82c0"
server_key_iv: "0df864bc7d3f7126f6cbf4a67ec4ab03"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.353902 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:24.365953 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:24.368848 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:24.369038 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:24.369491 3024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:24.370180 3024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:24.370229 3024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.370263 3024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:24.370283 3024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.379002 3024 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34865
I20260924 10:44:24.379029 3152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34865 every 8 connection(s)
I20260924 10:44:24.379856 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:24.388453 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3024
I20260924 10:44:24.388602 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:24.388924 2776 external_mini_cluster.cc:1428] Setting key 072c69f6636fe4f1a5af1bd2d3f1a8ea
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:24.393114 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.381699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:44843 (local address 127.2.182.62:37833)
0924 10:44:24.381835 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:24.381846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:24.382478 (+ 632us) server_negotiation.cc:407] Connection header received
0924 10:44:24.383892 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:24.383896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:24.383957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:24.384078 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:24.385287 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.385805 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.386881 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.387106 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.387213 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.389663 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:24.389667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:24.389670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:24.389694 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:24.391130 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.391625 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.391628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.391632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.391726 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.392082 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.392087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.392091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.392313 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:24.392416 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:24.392862 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 10:44:24.393000 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:44:24.393841 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.381921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37833 (local address 127.2.182.2:44843)
0924 10:44:24.382328 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:24.382367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:24.383617 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:24.384251 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:24.384257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:24.384613 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:24.385145 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.385159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.385940 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.385946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.386758 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.386767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.387243 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.387246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.387401 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.387883 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:24.387897 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:24.389474 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:24.391255 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.391260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.391273 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.391518 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.391838 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.391843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.391849 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.391977 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.392411 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:24.392416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:24.392626 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 10:44:24.392777 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 10:44:24.393086 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260924 10:44:24.394904 3153 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37833
I20260924 10:44:24.395134 3153 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:24.395663 3153 heartbeater.cc:511] Master 127.2.182.62:37833 requested a full tablet report, sending...
I20260924 10:44:24.396698 2828 ts_manager.cc:194] Registered new tserver with Master: 5a246e6ffa9c41e484b4fb0535318db4 (127.2.182.2:34865)
I20260924 10:44:24.397311 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:44843
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:24.415457 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:24.522549 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:24.522835 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:24.522917 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:24.526876 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:24.526963 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:24.527064 3158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:24.532310 3158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:24.533345 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:24.534327 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:24.543884 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:24.543890 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:24.543898 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:24.543957 3158 server_base.cc:1034] running on GCE node
I20260924 10:44:24.544530 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:24.545184 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:24.546366 3158 hybrid_clock.cc:648] HybridClock initialized: now 1790246664546352 us; error 26 us; skew 500 ppm
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:24.548991 3158 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:24.549835 3158 webserver.cc:460] Webserver started at http://127.2.182.3:45989/ using document root <none> and password file <none>
I20260924 10:44:24.550366 3158 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:24.550452 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:24.550681 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:24.552634 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "9640d90d729e45a4870d00be8a221074"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "cc9fdcf3a3cd030148ccac1a2f42ae03"
server_key_iv: "d1b5ade3d93e74f237c3902b627972c1"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.553211 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "9640d90d729e45a4870d00be8a221074"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "cc9fdcf3a3cd030148ccac1a2f42ae03"
server_key_iv: "d1b5ade3d93e74f237c3902b627972c1"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.556758 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:24.559283 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.560441 3158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:24.560566 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "9640d90d729e45a4870d00be8a221074"
format_stamp: "Formatted at 2026-09-24 10:44:24 on dist-test-slave-6lwt"
server_key: "cc9fdcf3a3cd030148ccac1a2f42ae03"
server_key_iv: "d1b5ade3d93e74f237c3902b627972c1"
server_key_version: "encryptionkey@0"
I20260924 10:44:24.560675 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:24.582468 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:24.585057 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:24.585239 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:24.585752 3158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:24.586486 3158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:24.586553 3158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.586627 3158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:24.586678 3158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:24.595058 3158 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:42191
I20260924 10:44:24.595095 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:42191 every 8 connection(s)
I20260924 10:44:24.595878 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:24.600940 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3158
I20260924 10:44:24.601042 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:24.601306 2776 external_mini_cluster.cc:1428] Setting key e6b5f6d989e7292b62e6863005688429
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:24.608970 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.597601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:54411 (local address 127.2.182.62:37833)
0924 10:44:24.597747 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:24.597750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:24.598534 (+ 784us) server_negotiation.cc:407] Connection header received
0924 10:44:24.599815 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:24.599818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:24.599872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:24.599969 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:24.601227 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.601873 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.603414 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.603680 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.603783 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.606033 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:24.606036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:24.606038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:24.606057 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:24.607299 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.607733 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.607735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.607737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.607821 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.608101 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.608103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.608104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.608256 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:24.608323 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:24.608719 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 10:44:24.608804 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:44:24.609719 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.597949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37833 (local address 127.2.182.3:54411)
0924 10:44:24.598380 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:24.598436 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:24.599613 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:24.600158 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:24.600164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:24.600485 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:24.601017 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.601040 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.601977 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.601981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.603303 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:24.603315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.603823 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.603826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:24.603961 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.604538 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:24.604553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:24.605887 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:24.607386 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.607391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.607401 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.607655 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.607896 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:24.607899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:24.607900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:24.608023 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:24.608331 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:24.608336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:24.608536 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 10:44:24.608676 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:44:24.608957 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260924 10:44:24.610765 3287 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37833
I20260924 10:44:24.610976 3287 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:24.611445 3287 heartbeater.cc:511] Master 127.2.182.62:37833 requested a full tablet report, sending...
I20260924 10:44:24.612353 2828 ts_manager.cc:194] Registered new tserver with Master: 9640d90d729e45a4870d00be8a221074 (127.2.182.3:42191)
I20260924 10:44:24.612929 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:54411
I20260924 10:44:24.615195 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:24.640630 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.631631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.2.182.1:40979)
0924 10:44:24.631994 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:24.632012 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:24.632036 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:44:24.632088 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:24.632093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:24.632279 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:24.632397 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:24.633040 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.633796 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.634590 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.634937 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.635079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.637370 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:24.637378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:24.637390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:24.637411 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:24.639092 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.639472 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.639477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.639478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.639550 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.639759 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.639761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.639762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.639964 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:24.640113 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:24.640282 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:44:24.640435 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":181,"threads_started":1}
W20260924 10:44:24.641609 2933 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:24.665343 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:24.657835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.2.182.1:40979)
0924 10:44:24.658016 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:24.658025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:24.658102 (+ 77us) server_negotiation.cc:407] Connection header received
0924 10:44:24.658202 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:24.658204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:24.658246 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:24.658311 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:24.658985 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.659479 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.660223 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:24.660433 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:24.660506 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:24.662710 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:24.662713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:24.662715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:24.662741 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:24.663921 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.664304 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.664306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.664308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.664372 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:24.664610 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:24.664612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:24.664613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:24.664760 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:24.664851 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:24.665053 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 10:44:24.665159 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:44:24.666283 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 2890
I20260924 10:44:24.672011 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3024
I20260924 10:44:24.678761 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3158
I20260924 10:44:24.684008 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 2796
2026-09-24T10:44:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2083 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[3301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[3301](info): set up 2 sockets
Sep 24 10:44:24 dist-test-slave-6lwt krb5kdc[3301](info): commencing operation
krb5kdc: starting...
W20260924 10:44:25.700632 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:25 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:25.816175 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35641
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:25.922123 3314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:25.922401 3314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:25.922482 3314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:25.926445 3314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:25.926523 3314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:25.926561 3314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:25.926620 3314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:25.926661 3314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:25.931715 3314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:25.932827 3314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:25.933848 3314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:25.941994 3314 server_base.cc:1034] running on GCE node
W20260924 10:44:25.941973 3320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:25.941942 3319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:25.941957 3322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:25.942778 3314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:25.943821 3314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:25.944964 3314 hybrid_clock.cc:648] HybridClock initialized: now 1790246665944946 us; error 28 us; skew 500 ppm
Sep 24 10:44:25 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246665, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:25.947538 3314 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:25.948380 3314 webserver.cc:460] Webserver started at http://127.2.182.62:43645/ using document root <none> and password file <none>
I20260924 10:44:25.948868 3314 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:25.948985 3314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:25.949154 3314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:25.950999 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "c941f5bac7c847229d2c88a50b13a6be"
format_stamp: "Formatted at 2026-09-24 10:44:25 on dist-test-slave-6lwt"
server_key: "11a0ac057d713599f65098aa82b1ee40"
server_key_iv: "722365592510fe7e7db51847fe51073f"
server_key_version: "encryptionkey@0"
I20260924 10:44:25.951479 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "c941f5bac7c847229d2c88a50b13a6be"
format_stamp: "Formatted at 2026-09-24 10:44:25 on dist-test-slave-6lwt"
server_key: "11a0ac057d713599f65098aa82b1ee40"
server_key_iv: "722365592510fe7e7db51847fe51073f"
server_key_version: "encryptionkey@0"
I20260924 10:44:25.954893 3314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:25.957041 3329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:25.958176 3314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:25.958266 3314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "c941f5bac7c847229d2c88a50b13a6be"
format_stamp: "Formatted at 2026-09-24 10:44:25 on dist-test-slave-6lwt"
server_key: "11a0ac057d713599f65098aa82b1ee40"
server_key_iv: "722365592510fe7e7db51847fe51073f"
server_key_version: "encryptionkey@0"
I20260924 10:44:25.958345 3314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:25.972236 3314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:25.974746 3314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:25.974900 3314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:25.981602 3314 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:35641
I20260924 10:44:25.981626 3381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:35641 every 8 connection(s)
I20260924 10:44:25.982491 3314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:25.985200 3382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:25.990355 3382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be: Bootstrap starting.
I20260924 10:44:25.991496 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3314
I20260924 10:44:25.991618 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:25.991873 2776 external_mini_cluster.cc:1428] Setting key 3b8a862f575b1fb3dc7ab280a89bc46a
I20260924 10:44:25.993176 3382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:25.993897 3382 log.cc:824] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:25.995899 3382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be: No bootstrap required, opened a new log
Sep 24 10:44:25 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:26.000329 3382 raft_consensus.cc:348] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } }
I20260924 10:44:26.000591 3382 raft_consensus.cc:374] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:26.000682 3382 raft_consensus.cc:729] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c941f5bac7c847229d2c88a50b13a6be, State: Initialized, Role: FOLLOWER
I20260924 10:44:26.001185 3382 consensus_queue.cc:260] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } }
I20260924 10:44:26.001334 3382 raft_consensus.cc:388] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:26.001403 3382 raft_consensus.cc:482] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:26.001497 3382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:26.002334 3382 raft_consensus.cc:504] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } }
I20260924 10:44:26.002656 3382 leader_election.cc:302] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c941f5bac7c847229d2c88a50b13a6be; no voters:
I20260924 10:44:26.002974 3382 leader_election.cc:288] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:26.003082 3387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:26.003296 3387 raft_consensus.cc:686] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [term 1 LEADER]: Becoming Leader. State: Replica: c941f5bac7c847229d2c88a50b13a6be, State: Running, Role: LEADER
I20260924 10:44:26.004076 3387 consensus_queue.cc:237] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } }
I20260924 10:44:26.004181 3382 sys_catalog.cc:564] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:26.005645 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [sys.catalog]: SysCatalogTable state changed. Reason: New leader c941f5bac7c847229d2c88a50b13a6be. Latest consensus state: current_term: 1 leader_uuid: "c941f5bac7c847229d2c88a50b13a6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } } }
I20260924 10:44:26.005788 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:26.006105 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c941f5bac7c847229d2c88a50b13a6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c941f5bac7c847229d2c88a50b13a6be" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35641 } } }
I20260924 10:44:26.006189 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:26.007031 3399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:26.007454 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:25.993008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.2.182.62:35641)
0924 10:44:25.993402 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:25.993420 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:25.993454 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:25.994088 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:25.994106 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:25.994422 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:25.994681 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:25.995473 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:25.996484 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:25.997647 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:25.998117 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:25.998236 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.001206 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.001211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.001229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.001254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.003387 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.003864 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.003870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.003875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.004011 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.004345 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.004348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.004350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.004876 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.005102 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.005950 (+ 848us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.006258 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260924 10:44:26.010149 3399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:26.015609 3399 catalog_manager.cc:1347] Generated new cluster ID: 5e9b3aded832465185a7c786b11f5dcc
I20260924 10:44:26.015667 3399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:26.033171 3399 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:26.034431 3399 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:26.044298 3399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c941f5bac7c847229d2c88a50b13a6be: Generated new TSK 0
I20260924 10:44:26.044986 3399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:26.076293 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:26.203671 3408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:26.203908 3408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:26.203949 3408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:26.207929 3408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:26.207988 3408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:26.208045 3408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:26.213152 3408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:26.214099 3408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:26.215047 3408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:26.224143 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:26.224191 3408 server_base.cc:1034] running on GCE node
W20260924 10:44:26.224352 3414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:26.224161 3413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:26.224823 3408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:26.225432 3408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:26.226584 3408 hybrid_clock.cc:648] HybridClock initialized: now 1790246666226569 us; error 27 us; skew 500 ppm
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:26.229200 3408 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:26.229957 3408 webserver.cc:460] Webserver started at http://127.2.182.1:38081/ using document root <none> and password file <none>
I20260924 10:44:26.230443 3408 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:26.230499 3408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:26.230650 3408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:26.232563 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "20b68452f4b6430e849a43efea73395b"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "6d19592877d8d192581840ddf00a8ed4"
server_key_iv: "bd0efaf9b49d5acc6fb6add7ec1d18dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.233173 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "20b68452f4b6430e849a43efea73395b"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "6d19592877d8d192581840ddf00a8ed4"
server_key_iv: "bd0efaf9b49d5acc6fb6add7ec1d18dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.236600 3408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:44:26.238868 3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.240028 3408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:26.240136 3408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "20b68452f4b6430e849a43efea73395b"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "6d19592877d8d192581840ddf00a8ed4"
server_key_iv: "bd0efaf9b49d5acc6fb6add7ec1d18dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.240237 3408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:26.251802 3408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:26.254410 3408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:26.254585 3408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:26.255085 3408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:26.255785 3408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:26.255828 3408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.255889 3408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:26.255919 3408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.264434 3408 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40619
I20260924 10:44:26.264454 3536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40619 every 8 connection(s)
I20260924 10:44:26.265230 3408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:26.272807 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3408
I20260924 10:44:26.272938 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:26.273242 2776 external_mini_cluster.cc:1428] Setting key 473373025df2fbb872326af7da20a4fe
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:26.277899 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.266818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:33399 (local address 127.2.182.62:35641)
0924 10:44:26.266944 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.266948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.267656 (+ 708us) server_negotiation.cc:407] Connection header received
0924 10:44:26.268914 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.268918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.268974 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.269078 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:26.270145 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.270662 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.271725 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.271948 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.272052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.274539 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.274542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.274545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.274589 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.275930 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.276450 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.276455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.276456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.276533 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.276854 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.276858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.276860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.277067 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.277169 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.277631 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.277742 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:26.278573 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.267102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:35641 (local address 127.2.182.1:33399)
0924 10:44:26.267511 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.267550 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.268704 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.269237 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.269243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.269579 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:26.269999 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.270010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.270779 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.270784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.271602 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.271611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.272084 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.272087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.272247 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.272833 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:26.272855 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:26.274389 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:26.276059 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.276065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.276077 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.276350 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.276622 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.276627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.276629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.276763 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.277177 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:26.277185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:26.277404 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.277588 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.277875 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260924 10:44:26.279639 3537 heartbeater.cc:347] Connected to a master server at 127.2.182.62:35641
I20260924 10:44:26.279850 3537 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:26.280306 3537 heartbeater.cc:511] Master 127.2.182.62:35641 requested a full tablet report, sending...
I20260924 10:44:26.281786 3346 ts_manager.cc:194] Registered new tserver with Master: 20b68452f4b6430e849a43efea73395b (127.2.182.1:40619)
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:26.282902 3346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:33399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:26.299512 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:26.405445 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:26.405725 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:26.405807 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:26.409813 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:26.409893 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:26.409994 3542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:26.415169 3542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:26.416185 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:26.417229 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:26.426625 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:26.426625 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:26.426641 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:26.426924 3542 server_base.cc:1034] running on GCE node
I20260924 10:44:26.427323 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:26.427899 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:26.429040 3542 hybrid_clock.cc:648] HybridClock initialized: now 1790246666429029 us; error 25 us; skew 500 ppm
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:26.431818 3542 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:26.432660 3542 webserver.cc:460] Webserver started at http://127.2.182.2:37723/ using document root <none> and password file <none>
I20260924 10:44:26.433532 3542 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:26.433646 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:26.433905 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:26.435959 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "183b21506671402a87649fe488653667"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "41f8178e46ff10934606cadb3763a388"
server_key_iv: "587a604f3708067ef2d3d35f9591656f"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.436511 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "183b21506671402a87649fe488653667"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "41f8178e46ff10934606cadb3763a388"
server_key_iv: "587a604f3708067ef2d3d35f9591656f"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.439967 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:26.442174 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.443287 3542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:26.443418 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "183b21506671402a87649fe488653667"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "41f8178e46ff10934606cadb3763a388"
server_key_iv: "587a604f3708067ef2d3d35f9591656f"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.443518 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:26.470773 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:26.474038 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:26.474241 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:26.474819 3542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:26.475626 3542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:26.475694 3542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.475759 3542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:26.475800 3542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.484463 3542 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:46575
I20260924 10:44:26.484493 3670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:46575 every 8 connection(s)
I20260924 10:44:26.485352 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:26.495512 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3542
I20260924 10:44:26.495654 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:26.495939 2776 external_mini_cluster.cc:1428] Setting key 6bd23da46cd53ab96c2ce0f11d4989a2
I20260924 10:44:26.498160 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.487057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:56627 (local address 127.2.182.62:35641)
0924 10:44:26.487193 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.487197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.487943 (+ 746us) server_negotiation.cc:407] Connection header received
0924 10:44:26.489309 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.489311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.489357 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.489429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:26.490488 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.490987 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.492021 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.492224 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.492298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.494571 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.494573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.494575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.494596 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.496025 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.496607 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.496610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.496612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.496719 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.497087 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.497090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.497092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.497311 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.497430 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.497887 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.498002 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:26.498881 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.487375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:35641 (local address 127.2.182.2:56627)
0924 10:44:26.487794 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.487835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.489121 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.489587 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.489593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.489921 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:26.490370 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.490383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.491089 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.491091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.491915 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.491923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.492335 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.492337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.492469 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.493029 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:26.493044 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:26.494426 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:26.496155 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.496161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.496175 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.496482 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.496812 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.496814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.496815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.496999 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.497436 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:26.497444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:26.497667 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.497833 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.498131 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260924 10:44:26.500092 3671 heartbeater.cc:347] Connected to a master server at 127.2.182.62:35641
I20260924 10:44:26.500324 3671 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:26.500797 3671 heartbeater.cc:511] Master 127.2.182.62:35641 requested a full tablet report, sending...
I20260924 10:44:26.501742 3346 ts_manager.cc:194] Registered new tserver with Master: 183b21506671402a87649fe488653667 (127.2.182.2:46575)
I20260924 10:44:26.502326 3346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:56627
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:26.523128 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:26.630321 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:26.630587 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:26.630663 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:26.634696 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:26.634744 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:26.634846 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:26.639976 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:26.640980 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:26.641908 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:26.651065 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:26.651026 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:26.651046 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:26.651093 3676 server_base.cc:1034] running on GCE node
I20260924 10:44:26.651649 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:26.652247 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:26.653422 3676 hybrid_clock.cc:648] HybridClock initialized: now 1790246666653404 us; error 34 us; skew 500 ppm
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:26.656006 3676 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:26.656836 3676 webserver.cc:460] Webserver started at http://127.2.182.3:43593/ using document root <none> and password file <none>
I20260924 10:44:26.657395 3676 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:26.657452 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:26.657653 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:26.659663 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "f237c581690045f4881d0615e1579a7f"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "37e0e73d808b89856bfa79ed61346101"
server_key_iv: "d7fd5f33cc5a7c3d59fe87c3943a42fb"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.660205 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "f237c581690045f4881d0615e1579a7f"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "37e0e73d808b89856bfa79ed61346101"
server_key_iv: "d7fd5f33cc5a7c3d59fe87c3943a42fb"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.663844 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:26.666131 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.667198 3676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:26.667286 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "f237c581690045f4881d0615e1579a7f"
format_stamp: "Formatted at 2026-09-24 10:44:26 on dist-test-slave-6lwt"
server_key: "37e0e73d808b89856bfa79ed61346101"
server_key_iv: "d7fd5f33cc5a7c3d59fe87c3943a42fb"
server_key_version: "encryptionkey@0"
I20260924 10:44:26.667397 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:26.685946 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:26.688534 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:26.688716 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:26.689335 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:26.690093 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:26.690137 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.690208 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:26.690244 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:26.698716 3676 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:43923
I20260924 10:44:26.698745 3804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:43923 every 8 connection(s)
I20260924 10:44:26.699584 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:26.708886 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3676
I20260924 10:44:26.709052 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:26.709333 2776 external_mini_cluster.cc:1428] Setting key 1dcacd17aaa1a3af41d053c74b1e4b2b
I20260924 10:44:26.712821 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.701302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:54841 (local address 127.2.182.62:35641)
0924 10:44:26.701428 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.701431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.702181 (+ 750us) server_negotiation.cc:407] Connection header received
0924 10:44:26.703502 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.703505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.703563 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.703651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:26.704857 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.705503 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.706692 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.706914 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.707022 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.709551 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.709554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.709556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.709578 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.711098 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.711636 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.711639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.711640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.711711 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.711994 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.711996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.711997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.712154 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.712225 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.712610 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.712698 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:44:26.713641 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.701620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:35641 (local address 127.2.182.3:54841)
0924 10:44:26.702031 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.702071 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.703291 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.703825 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.703832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.704178 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:26.704724 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.704739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.705620 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.705625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.706578 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.706587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.707054 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.707058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.707228 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.707809 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:26.707824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:26.709345 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:26.711233 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.711237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.711248 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.711551 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.711795 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:26.711797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:26.711798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:26.711920 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:26.712217 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:26.712221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:26.712424 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.712566 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.712790 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260924 10:44:26.714661 3805 heartbeater.cc:347] Connected to a master server at 127.2.182.62:35641
I20260924 10:44:26.714881 3805 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:26.715338 3805 heartbeater.cc:511] Master 127.2.182.62:35641 requested a full tablet report, sending...
I20260924 10:44:26.716255 3346 ts_manager.cc:194] Registered new tserver with Master: f237c581690045f4881d0615e1579a7f (127.2.182.3:43923)
I20260924 10:44:26.716792 3346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:54841
I20260924 10:44:26.722836 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:44:26.733168 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.725999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.2.182.62:35641)
0924 10:44:26.726132 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.726136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.726288 (+ 152us) server_negotiation.cc:407] Connection header received
0924 10:44:26.726428 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.726431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.726474 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.726554 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:26.727232 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.727721 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.728537 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.728750 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.728855 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.730151 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.730154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.730157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.730187 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.731558 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.731963 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.731968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.731970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.732045 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.732326 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.732329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.732331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.732524 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.732658 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.732887 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.733016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:44:26.737646 3346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:26.740485 3346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:26.754411 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.749830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.2.182.3:43923)
0924 10:44:26.750186 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.750198 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.750221 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:44:26.750294 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.750298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.750459 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.750605 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:26.750772 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751839 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.752866 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753702 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753841 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.753992 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.754162 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.754252 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
I20260924 10:44:26.754431 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.748511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40619 (local address 127.0.0.1:44666)
0924 10:44:26.748944 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.748992 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.749127 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.749910 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.749915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.749936 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:26.750074 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.750081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751390 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.751397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.753190 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.753196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753991 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.754087 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.754126 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.754167 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.754292 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":13,"thread_start_us":79,"threads_started":1}
I20260924 10:44:26.754670 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.749233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:43923 (local address 127.0.0.1:45042)
0924 10:44:26.749458 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.749472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.749549 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.750583 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.750585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.750609 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:26.750684 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.750690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751957 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.751961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.752747 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.752755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753843 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.753973 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.754016 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.754121 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.754550 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":26,"thread_start_us":91,"threads_started":1}
I20260924 10:44:26.754731 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.748629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.2.182.1:40619)
0924 10:44:26.749412 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.749427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.749449 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:44:26.749531 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.749536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.749744 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.750238 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:26.750385 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751264 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753297 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753866 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.754321 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.754456 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.754505 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.754587 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":87,"threads_started":1}
I20260924 10:44:26.754784 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.749314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.2.182.2:46575)
0924 10:44:26.749829 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.749840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.749861 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:44:26.749941 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.749947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.750127 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.750330 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:26.750513 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751663 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753030 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753782 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.754364 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.754509 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.754568 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.754647 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":102,"threads_started":1}
I20260924 10:44:26.754993 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.748574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:46575 (local address 127.0.0.1:48288)
0924 10:44:26.749082 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.749094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.749158 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.750310 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.750313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.750329 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:26.750414 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.750422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.751802 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.751806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.752909 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.752920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.753909 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.753912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.753996 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.754024 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.754127 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.754263 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20260924 10:44:26.757571 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 47125d54fd37453984657df737921ca6 (DEFAULT_TABLE table=test-table [id=8603a6ce3c844ce88b518c8bd88987f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:26.757716 3739 tablet_service.cc:1467] Processing CreateTablet for tablet 47125d54fd37453984657df737921ca6 (DEFAULT_TABLE table=test-table [id=8603a6ce3c844ce88b518c8bd88987f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:26.757716 3605 tablet_service.cc:1467] Processing CreateTablet for tablet 47125d54fd37453984657df737921ca6 (DEFAULT_TABLE table=test-table [id=8603a6ce3c844ce88b518c8bd88987f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:26.758526 3471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47125d54fd37453984657df737921ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:26.758665 3605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47125d54fd37453984657df737921ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:26.758665 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47125d54fd37453984657df737921ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:26.764966 3822 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Bootstrap starting.
I20260924 10:44:26.766002 3824 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Bootstrap starting.
I20260924 10:44:26.767397 3822 tablet_bootstrap.cc:654] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:26.768059 3824 tablet_bootstrap.cc:654] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:26.768587 3822 log.cc:824] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:26.769176 3824 log.cc:824] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:26.769225 3823 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Bootstrap starting.
I20260924 10:44:26.770622 3822 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: No bootstrap required, opened a new log
I20260924 10:44:26.770869 3822 ts_tablet_manager.cc:1397] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260924 10:44:26.771294 3824 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: No bootstrap required, opened a new log
I20260924 10:44:26.771500 3824 ts_tablet_manager.cc:1397] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:44:26.771692 3823 tablet_bootstrap.cc:654] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:26.772451 3823 log.cc:824] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:26.774140 3822 raft_consensus.cc:348] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.774302 3823 tablet_bootstrap.cc:492] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: No bootstrap required, opened a new log
I20260924 10:44:26.774361 3822 raft_consensus.cc:374] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:26.774431 3822 raft_consensus.cc:729] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b68452f4b6430e849a43efea73395b, State: Initialized, Role: FOLLOWER
I20260924 10:44:26.774457 3823 ts_tablet_manager.cc:1397] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:44:26.774869 3822 consensus_queue.cc:260] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.775297 3824 raft_consensus.cc:348] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.775504 3537 heartbeater.cc:503] Master 127.2.182.62:35641 was elected leader, sending a full tablet report...
I20260924 10:44:26.775547 3824 raft_consensus.cc:374] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:26.775664 3824 raft_consensus.cc:729] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f237c581690045f4881d0615e1579a7f, State: Initialized, Role: FOLLOWER
I20260924 10:44:26.775844 3822 ts_tablet_manager.cc:1428] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 10:44:26.776089 3824 consensus_queue.cc:260] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.776669 3805 heartbeater.cc:503] Master 127.2.182.62:35641 was elected leader, sending a full tablet report...
I20260924 10:44:26.776996 3824 ts_tablet_manager.cc:1428] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:44:26.781214 3823 raft_consensus.cc:348] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.781448 3823 raft_consensus.cc:374] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:26.781517 3823 raft_consensus.cc:729] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183b21506671402a87649fe488653667, State: Initialized, Role: FOLLOWER
I20260924 10:44:26.781860 3823 consensus_queue.cc:260] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.782449 3671 heartbeater.cc:503] Master 127.2.182.62:35641 was elected leader, sending a full tablet report...
I20260924 10:44:26.782696 3823 ts_tablet_manager.cc:1428] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:44:26.920439 3828 raft_consensus.cc:482] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:26.920696 3828 raft_consensus.cc:504] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.921826 3828 leader_election.cc:288] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 183b21506671402a87649fe488653667 (127.2.182.2:46575), f237c581690045f4881d0615e1579a7f (127.2.182.3:43923)
I20260924 10:44:26.925832 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.922189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:60787 (local address 127.2.182.3:43923)
0924 10:44:26.922353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.922358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.922528 (+ 170us) server_negotiation.cc:407] Connection header received
0924 10:44:26.922635 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.922638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.922685 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.922782 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:26.923129 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.923684 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.924843 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.925309 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.925422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.925480 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.925646 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.925707 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 10:44:26.925854 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.922128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:39237 (local address 127.2.182.2:46575)
0924 10:44:26.922354 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.922359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.922409 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:44:26.922573 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.922576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.922645 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.922744 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:26.923116 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.923713 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.924933 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.925325 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.925436 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.925495 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.925679 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.925738 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 10:44:26.925894 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.922030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:46575 (local address 127.2.182.1:39237)
0924 10:44:26.922315 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.922333 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.922443 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.922786 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.922788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.922837 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:26.922980 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.922986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.923821 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.923825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.924786 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.924795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.925438 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.925441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.925547 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.925566 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.925664 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.925732 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"spinlock_wait_cycles":9216,"thread_start_us":87,"threads_started":1}
I20260924 10:44:26.925902 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.922045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:43923 (local address 127.2.182.1:60787)
0924 10:44:26.922443 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:26.922455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:26.922535 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:26.922786 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:26.922788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:26.922833 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:26.923013 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.923020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.923793 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.923798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.924720 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:26.924728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.925425 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.925428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:26.925520 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.925542 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:44:26.925634 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:44:26.925699 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":75,"spinlock_wait_cycles":768,"thread_start_us":142,"threads_started":1}
I20260924 10:44:26.926584 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47125d54fd37453984657df737921ca6" candidate_uuid: "20b68452f4b6430e849a43efea73395b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183b21506671402a87649fe488653667" is_pre_election: true
I20260924 10:44:26.926783 3759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47125d54fd37453984657df737921ca6" candidate_uuid: "20b68452f4b6430e849a43efea73395b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f237c581690045f4881d0615e1579a7f" is_pre_election: true
I20260924 10:44:26.926913 3625 raft_consensus.cc:2454] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b68452f4b6430e849a43efea73395b in term 0.
I20260924 10:44:26.927024 3759 raft_consensus.cc:2454] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b68452f4b6430e849a43efea73395b in term 0.
I20260924 10:44:26.927366 3426 leader_election.cc:302] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183b21506671402a87649fe488653667, 20b68452f4b6430e849a43efea73395b; no voters:
I20260924 10:44:26.927639 3828 raft_consensus.cc:2781] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:26.927721 3828 raft_consensus.cc:482] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:26.927774 3828 raft_consensus.cc:3037] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:26.928753 3828 raft_consensus.cc:504] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.929173 3828 leader_election.cc:288] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [CANDIDATE]: Term 1 election: Requested vote from peers 183b21506671402a87649fe488653667 (127.2.182.2:46575), f237c581690045f4881d0615e1579a7f (127.2.182.3:43923)
I20260924 10:44:26.929451 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47125d54fd37453984657df737921ca6" candidate_uuid: "20b68452f4b6430e849a43efea73395b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183b21506671402a87649fe488653667"
I20260924 10:44:26.929510 3759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47125d54fd37453984657df737921ca6" candidate_uuid: "20b68452f4b6430e849a43efea73395b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f237c581690045f4881d0615e1579a7f"
I20260924 10:44:26.929579 3625 raft_consensus.cc:3037] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:26.929675 3759 raft_consensus.cc:3037] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:26.930605 3625 raft_consensus.cc:2454] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b68452f4b6430e849a43efea73395b in term 1.
I20260924 10:44:26.930739 3759 raft_consensus.cc:2454] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b68452f4b6430e849a43efea73395b in term 1.
I20260924 10:44:26.930881 3426 leader_election.cc:302] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183b21506671402a87649fe488653667, 20b68452f4b6430e849a43efea73395b; no voters:
I20260924 10:44:26.931023 3828 raft_consensus.cc:2781] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:26.931272 3828 raft_consensus.cc:686] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [term 1 LEADER]: Becoming Leader. State: Replica: 20b68452f4b6430e849a43efea73395b, State: Running, Role: LEADER
I20260924 10:44:26.931572 3828 consensus_queue.cc:237] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } }
I20260924 10:44:26.934696 3346 catalog_manager.cc:5495] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b68452f4b6430e849a43efea73395b (127.2.182.1). New cstate: current_term: 1 leader_uuid: "20b68452f4b6430e849a43efea73395b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b68452f4b6430e849a43efea73395b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183b21506671402a87649fe488653667" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f237c581690045f4881d0615e1579a7f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43923 } health_report { overall_health: UNKNOWN } } }
W20260924 10:44:26.950426 3806 tablet.cc:2369] T 47125d54fd37453984657df737921ca6 P f237c581690045f4881d0615e1579a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:44:26.986392 3672 tablet.cc:2369] T 47125d54fd37453984657df737921ca6 P 183b21506671402a87649fe488653667: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:44:26 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:26.992728 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:26.982888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.2.182.1:40619)
0924 10:44:26.983028 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:26.983034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:26.983202 (+ 168us) server_negotiation.cc:407] Connection header received
0924 10:44:26.983357 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:26.983360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:26.983416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:26.983501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:26.984194 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.984696 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.985526 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:26.985737 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:26.985843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:26.988715 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:26.988726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:26.988739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:26.988772 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:26.990652 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.991124 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.991134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.991139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.991247 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:26.991581 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:26.991586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:26.991589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:26.991941 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:26.992159 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:26.992330 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:44:26.992466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 10:44:27.016424 3538 tablet.cc:2369] T 47125d54fd37453984657df737921ca6 P 20b68452f4b6430e849a43efea73395b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:27.022734 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:27.015066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.2.182.1:40619)
0924 10:44:27.015220 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:27.015224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:27.015279 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:44:27.015387 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:27.015390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:27.015429 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:27.015496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:27.016074 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:27.016703 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:27.017546 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:27.017766 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:27.017836 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:27.020061 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:27.020064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:27.020066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:27.020084 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:27.021341 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:27.021683 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:27.021685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:27.021686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:27.021750 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:27.021977 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:27.021979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:27.021980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:27.022125 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:27.022225 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:27.022446 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 10:44:27.022530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:27.047585 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:27.039563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.2.182.1:40619)
0924 10:44:27.039703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:27.039707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:27.039917 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:44:27.040012 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:27.040015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:27.040056 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:27.040125 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:27.040713 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:27.041277 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:27.042036 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:27.042229 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:27.042302 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:27.044626 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:27.044628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:27.044630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:27.044651 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:27.046108 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:27.046516 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:27.046518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:27.046519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:27.046586 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:27.046849 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:27.046853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:27.046856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:27.047009 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:27.047123 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:27.047291 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 10:44:27.047397 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:44:27.048996 3451 tablet_service.cc:2908] Rejecting scan request for tablet 47125d54fd37453984657df737921ca6: Uninitialized: clean time has not yet been initialized
I20260924 10:44:27.050884 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3408
I20260924 10:44:27.056829 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3542
I20260924 10:44:27.062098 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3676
I20260924 10:44:27.067607 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3314
2026-09-24T10:44:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2383 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3847](info): set up 2 sockets
Sep 24 10:44:27 dist-test-slave-6lwt krb5kdc[3847](info): commencing operation
krb5kdc: starting...
W20260924 10:44:28.080557 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:28.215896 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:39759
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:28.324878 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:28.325145 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:28.325183 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:28.328969 3860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:28.329012 3860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:44:28.329047 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:28.329079 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:28.329095 3860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:28.329110 3860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:28.334163 3860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:39759
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:28.335036 3860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:28.335940 3860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:28.344017 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.344097 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.344131 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:28.344291 3860 server_base.cc:1034] running on GCE node
I20260924 10:44:28.344826 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:28.345865 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:28.347041 3860 hybrid_clock.cc:648] HybridClock initialized: now 1790246668347026 us; error 27 us; skew 500 ppm
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:28.349556 3860 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:28.350299 3860 webserver.cc:460] Webserver started at http://127.2.182.62:44749/ using document root <none> and password file <none>
I20260924 10:44:28.350811 3860 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:28.350885 3860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:28.351095 3860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:28.353114 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "20618ce53a444177a10091af89e60309"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "da549b271deb1da5b6cf74272c5088d9"
server_key_iv: "dec3babb35f96ee7b876329d94fdbbd6"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.353675 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "20618ce53a444177a10091af89e60309"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "da549b271deb1da5b6cf74272c5088d9"
server_key_iv: "dec3babb35f96ee7b876329d94fdbbd6"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.357219 3860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:44:28.359540 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.360737 3860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:28.360879 3860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "20618ce53a444177a10091af89e60309"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "da549b271deb1da5b6cf74272c5088d9"
server_key_iv: "dec3babb35f96ee7b876329d94fdbbd6"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.361054 3860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:28.385262 3860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:28.388123 3860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:28.388331 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:28.395985 3860 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:39759
I20260924 10:44:28.396009 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:39759 every 8 connection(s)
I20260924 10:44:28.396873 3860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:28.399505 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:28.401741 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3860
I20260924 10:44:28.401839 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:28.402107 2776 external_mini_cluster.cc:1428] Setting key f07eb10d37c1378f9ce55e0d067aa2f3
I20260924 10:44:28.405591 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Bootstrap starting.
I20260924 10:44:28.408151 3938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:28.408973 3938 log.cc:824] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:28.410734 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: No bootstrap required, opened a new log
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:28.414922 3938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } }
I20260924 10:44:28.415185 3938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:28.415266 3938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20618ce53a444177a10091af89e60309, State: Initialized, Role: FOLLOWER
I20260924 10:44:28.415747 3938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } }
I20260924 10:44:28.415907 3938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:28.415980 3938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:28.416075 3938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:28.417007 3938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } }
I20260924 10:44:28.417089 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.403480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.2.182.62:39759)
0924 10:44:28.403879 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:28.403897 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:28.403939 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:44:28.404748 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:28.404769 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:28.405204 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:28.405548 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:28.406234 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.407695 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.408652 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.409387 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.409569 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.412298 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:28.412303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:28.412315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:28.412340 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:28.414051 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.414530 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.414534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.414538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.414640 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.414918 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.414920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.414921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.415420 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:28.415610 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:28.415837 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 10:44:28.416298 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260924 10:44:28.417374 3938 leader_election.cc:302] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20618ce53a444177a10091af89e60309; no voters:
I20260924 10:44:28.417678 3938 leader_election.cc:288] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:28.417791 3943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:28.417979 3943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [term 1 LEADER]: Becoming Leader. State: Replica: 20618ce53a444177a10091af89e60309, State: Running, Role: LEADER
I20260924 10:44:28.418332 3943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } }
I20260924 10:44:28.418834 3938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:28.419878 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20618ce53a444177a10091af89e60309. Latest consensus state: current_term: 1 leader_uuid: "20618ce53a444177a10091af89e60309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } } }
I20260924 10:44:28.420004 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:28.420248 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20618ce53a444177a10091af89e60309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20618ce53a444177a10091af89e60309" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39759 } } }
I20260924 10:44:28.420347 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:28.421176 3952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:28.424069 3952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:28.429625 3952 catalog_manager.cc:1347] Generated new cluster ID: 00b87e072ee04307bb8186d19723f097
I20260924 10:44:28.429690 3952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:28.452245 3952 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:28.453270 3952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:28.469204 3952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Generated new TSK 0
I20260924 10:44:28.469916 3952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:28.504173 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:39759
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:28.609948 3964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:28.610255 3964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:28.610335 3964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:28.614283 3964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:28.614370 3964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:28.614439 3964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:28.614526 3964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:28.619626 3964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:28.620731 3964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:28.621819 3964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:28.630702 3972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.630788 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.630769 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:28.630996 3964 server_base.cc:1034] running on GCE node
I20260924 10:44:28.631386 3964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:28.631989 3964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:28.633159 3964 hybrid_clock.cc:648] HybridClock initialized: now 1790246668633130 us; error 46 us; skew 500 ppm
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:28.635648 3964 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:28.636379 3964 webserver.cc:460] Webserver started at http://127.2.182.1:37245/ using document root <none> and password file <none>
I20260924 10:44:28.636929 3964 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:28.637022 3964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:28.637236 3964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:28.639197 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "7fb6ee024078d39d8321b28eb9ebefe7"
server_key_iv: "366210aef72760f11a56c0df855f80e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.639798 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "7fb6ee024078d39d8321b28eb9ebefe7"
server_key_iv: "366210aef72760f11a56c0df855f80e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.643206 3964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:28.645412 3979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.646508 3964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:28.646649 3964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "7fb6ee024078d39d8321b28eb9ebefe7"
server_key_iv: "366210aef72760f11a56c0df855f80e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.646754 3964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:28.658783 3964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:28.661338 3964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:28.661520 3964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:28.662776 3964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:28.662859 3964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.662921 3964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:28.662973 3964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:28.673318 3964 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40831
I20260924 10:44:28.673425 4094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40831 every 8 connection(s)
I20260924 10:44:28.674346 3964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:28.676652 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.663579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:52731 (local address 127.2.182.62:39759)
0924 10:44:28.663718 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:28.663723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:28.664543 (+ 820us) server_negotiation.cc:407] Connection header received
0924 10:44:28.665659 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:28.665663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:28.665725 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:28.665803 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:28.667414 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.668087 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.669735 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.670043 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.670158 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.673186 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:28.673188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:28.673190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:28.673211 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:28.674665 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.675229 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.675237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.675238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.675307 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.675624 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.675627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.675629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.675799 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:28.675913 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:28.676375 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 10:44:28.676490 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:44:28.677379 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.663931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.1:52731)
0924 10:44:28.664389 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:28.664436 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:28.665418 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:28.666208 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:28.666216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:28.666694 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:28.667276 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.667294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.668212 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.668216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.669626 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.669638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.670197 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.670201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.670367 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.671499 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:28.671521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:28.672995 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:28.674798 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.674804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.674816 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.675118 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.675394 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.675398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.675400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.675528 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.675927 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:28.675934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:28.676145 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 10:44:28.676319 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 10:44:28.676596 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":174,"threads_started":1}
I20260924 10:44:28.678458 4095 heartbeater.cc:347] Connected to a master server at 127.2.182.62:39759
I20260924 10:44:28.678599 4095 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:28.679078 4095 heartbeater.cc:511] Master 127.2.182.62:39759 requested a full tablet report, sending...
I20260924 10:44:28.680047 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 3964
I20260924 10:44:28.680181 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:28.680476 2776 external_mini_cluster.cc:1428] Setting key 559cc4286a52f9b7a90b98a493c1c5cd
I20260924 10:44:28.680649 3891 ts_manager.cc:194] Registered new tserver with Master: 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831)
I20260924 10:44:28.681912 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:52731
I20260924 10:44:28.689885 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.680308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.1:43537)
0924 10:44:28.680574 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:28.680588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:28.680668 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:28.682798 (+ 2130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:28.682800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:28.682944 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:28.683028 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.683033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.684152 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.684157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.685143 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.685153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.685765 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.685767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.685832 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.686287 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:28.686308 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:28.686585 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:28.688458 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.688461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.688463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.688637 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.688968 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.688971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.688972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.689045 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.689468 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:28.689470 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:28.689543 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 10:44:28.689624 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:44:28.689718 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260924 10:44:28.690033 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.682406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:43537 (local address 127.2.182.62:39759)
0924 10:44:28.682523 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:28.682527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:28.682539 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:28.682570 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:28.682575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:28.682613 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:28.682692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:28.683184 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.683982 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.685265 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.685657 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.685784 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.686725 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:28.686732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:28.686738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:28.686771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:28.688347 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.688728 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.688733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.688736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.688827 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.689141 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.689146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.689148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.689364 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:28.689469 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:28.689714 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 10:44:28.689830 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:28.708266 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:39759
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:28.816365 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:28.816586 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:28.816632 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:28.820528 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:28.820578 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:28.820641 4104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:28.820703 4104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:28.825709 4104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:28.826577 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:28.827643 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:28.838725 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.838725 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:28.838759 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:28.838896 4104 server_base.cc:1034] running on GCE node
I20260924 10:44:28.839397 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:28.839985 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:28.841161 4104 hybrid_clock.cc:648] HybridClock initialized: now 1790246668841143 us; error 36 us; skew 500 ppm
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:28.843868 4104 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:28.844628 4104 webserver.cc:460] Webserver started at http://127.2.182.2:43621/ using document root <none> and password file <none>
I20260924 10:44:28.845191 4104 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:28.845253 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:28.845409 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:28.847256 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "49fc78eeb4244a2c93db2751a42c943d"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "6688fd36fd5fd58f20ef2966f30c4d47"
server_key_iv: "9832e5f690fdd6018834f46c0044f45c"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.847735 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "49fc78eeb4244a2c93db2751a42c943d"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "6688fd36fd5fd58f20ef2966f30c4d47"
server_key_iv: "9832e5f690fdd6018834f46c0044f45c"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.851181 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:28.853319 4119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.854432 4104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:28.854552 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "49fc78eeb4244a2c93db2751a42c943d"
format_stamp: "Formatted at 2026-09-24 10:44:28 on dist-test-slave-6lwt"
server_key: "6688fd36fd5fd58f20ef2966f30c4d47"
server_key_iv: "9832e5f690fdd6018834f46c0044f45c"
server_key_version: "encryptionkey@0"
I20260924 10:44:28.854646 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:28.886160 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:28.888847 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:28.889089 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:28.890705 4104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:28.890784 4104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.890854 4104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:28.890913 4104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:28.901849 4104 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:43195
I20260924 10:44:28.901947 4234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:43195 every 8 connection(s)
I20260924 10:44:28.903014 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:28.903690 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4104
I20260924 10:44:28.903820 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:28.904117 2776 external_mini_cluster.cc:1428] Setting key 4ca2d71cd775ffa50ac5034cd926676d
Sep 24 10:44:28 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:28.910193 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.891350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:51675 (local address 127.2.182.62:39759)
0924 10:44:28.891501 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:28.891507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:28.892688 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 10:44:28.893966 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:28.893969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:28.894026 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:28.894140 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:28.895594 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.896428 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.897708 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.897944 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.898061 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.906180 (+ 8119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:28.906187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:28.906193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:28.906227 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:28.907786 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.908334 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.908338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.908340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.908453 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.908849 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.908851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.908852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.909132 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:28.909220 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:28.909970 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 10:44:28.910081 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:28.910948 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.892112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.2:51675)
0924 10:44:28.892540 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:28.892597 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:28.893738 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:28.894307 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:28.894317 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:28.894713 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:28.895409 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.895428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.896581 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.896586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.897567 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.897603 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.898104 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.898108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.898274 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.904253 (+ 5979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:28.904269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:28.905969 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:28.907904 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.907910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.907923 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.908171 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.908550 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.908556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.908561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.908749 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.909240 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:28.909249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:28.909556 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 10:44:28.909777 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:44:28.910114 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":135,"threads_started":1}
I20260924 10:44:28.912230 4235 heartbeater.cc:347] Connected to a master server at 127.2.182.62:39759
I20260924 10:44:28.912369 4235 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:28.912917 4235 heartbeater.cc:511] Master 127.2.182.62:39759 requested a full tablet report, sending...
I20260924 10:44:28.913985 3892 ts_manager.cc:194] Registered new tserver with Master: 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195)
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:44:28.914676 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:51675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
I20260924 10:44:28.921828 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.914236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38199 (local address 127.2.182.62:39759)
0924 10:44:28.914355 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:28.914361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:28.914609 (+ 248us) server_negotiation.cc:407] Connection header received
0924 10:44:28.914750 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:28.914754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:28.914799 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:28.914880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:28.915277 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.915877 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.916863 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.917308 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.917427 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.918443 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:28.918446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:28.918449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:28.918470 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:28.920060 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.920531 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.920537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.920559 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.920647 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:28.920974 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:28.920978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:28.920980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:28.921225 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:28.921337 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:28.921596 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 10:44:28.921695 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:44:28.921840 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:28.914248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.2:38199)
0924 10:44:28.914529 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:28.914540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:28.914654 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:28.914894 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:28.914897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:28.915082 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:28.915171 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.915176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.916000 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.916003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.916747 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:28.916756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:28.917438 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:28.917442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:28.917543 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:28.918044 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:28.918055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:28.918319 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:28.920167 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.920169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.920171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.920423 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.920727 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:28.920729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:28.920731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:28.920855 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:28.921338 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:28.921341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:28.921428 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:44:28.921529 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:44:28.921659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:28.931877 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:39759
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:29.039909 4244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:29.040165 4244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:29.040238 4244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:29.044094 4244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:29.044174 4244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:29.044242 4244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:29.044322 4244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:29.049304 4244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:29.050275 4244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:29.051221 4244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:29.060439 4250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:29.060420 4244 server_base.cc:1034] running on GCE node
W20260924 10:44:29.060400 4249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:29.060410 4252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:29.061035 4244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:29.061609 4244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:29.062798 4244 hybrid_clock.cc:648] HybridClock initialized: now 1790246669062766 us; error 44 us; skew 500 ppm
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246669, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:29.065529 4244 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:29.066282 4244 webserver.cc:460] Webserver started at http://127.2.182.3:33989/ using document root <none> and password file <none>
I20260924 10:44:29.066813 4244 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:29.066901 4244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:29.067116 4244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:29.069034 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "9ea1cd291cf741d6adcdea2f1519d779"
format_stamp: "Formatted at 2026-09-24 10:44:29 on dist-test-slave-6lwt"
server_key: "3edf6d8c3e76713815164e090b281e2d"
server_key_iv: "d13aac68528b1002cf7393ac01319723"
server_key_version: "encryptionkey@0"
I20260924 10:44:29.069561 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea1cd291cf741d6adcdea2f1519d779"
format_stamp: "Formatted at 2026-09-24 10:44:29 on dist-test-slave-6lwt"
server_key: "3edf6d8c3e76713815164e090b281e2d"
server_key_iv: "d13aac68528b1002cf7393ac01319723"
server_key_version: "encryptionkey@0"
I20260924 10:44:29.072942 4244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:29.075073 4259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:29.076182 4244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:29.076308 4244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "9ea1cd291cf741d6adcdea2f1519d779"
format_stamp: "Formatted at 2026-09-24 10:44:29 on dist-test-slave-6lwt"
server_key: "3edf6d8c3e76713815164e090b281e2d"
server_key_iv: "d13aac68528b1002cf7393ac01319723"
server_key_version: "encryptionkey@0"
I20260924 10:44:29.076419 4244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:29.086472 4244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:29.089082 4244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:29.089262 4244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:29.090574 4244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:29.090641 4244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:29.090698 4244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:29.090754 4244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246669, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:29.101234 4244 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44953
I20260924 10:44:29.101328 4374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44953 every 8 connection(s)
I20260924 10:44:29.102228 4244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:29.104485 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.091514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:33117 (local address 127.2.182.62:39759)
0924 10:44:29.091657 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.091661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.092604 (+ 943us) server_negotiation.cc:407] Connection header received
0924 10:44:29.093709 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.093716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.093781 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.093881 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.095357 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.095875 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.097542 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.097774 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.097890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.100983 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.100987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.100990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.101016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.102635 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.103224 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.103231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.103235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.103318 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.103629 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.103631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.103632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.103802 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.103875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.104286 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.104374 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:44:29.105315 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.091991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.3:33117)
0924 10:44:29.092430 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.092467 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.093463 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.094100 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.094109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.094564 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:29.095159 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.095176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.096000 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.096003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.097420 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.097430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.097938 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.097941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.098118 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.099312 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:29.099350 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:29.100766 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:29.102737 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.102743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.102762 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.103099 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.103410 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.103412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.103414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.103540 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.103886 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:29.103890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:29.104087 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.104240 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.104456 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":137,"threads_started":1}
I20260924 10:44:29.106434 4375 heartbeater.cc:347] Connected to a master server at 127.2.182.62:39759
I20260924 10:44:29.106561 4375 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:29.107126 4375 heartbeater.cc:511] Master 127.2.182.62:39759 requested a full tablet report, sending...
I20260924 10:44:29.107336 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4244
I20260924 10:44:29.107451 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:29.107722 2776 external_mini_cluster.cc:1428] Setting key 14f547a6145c5b123f3c642321023407
I20260924 10:44:29.108201 3891 ts_manager.cc:194] Registered new tserver with Master: 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3:44953)
I20260924 10:44:29.108971 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:33117
I20260924 10:44:29.109854 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:44:29.116793 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.108707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.2.182.3:37843)
0924 10:44:29.108998 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.109014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.109138 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.109381 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.109383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.109534 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:29.109660 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.109667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.110673 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.110680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.111709 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.111724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.112325 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.112327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.112391 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.112940 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:29.112952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:29.113341 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:29.115266 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.115271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.115288 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.115545 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.115795 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.115801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.115806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.115946 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.116262 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:29.116268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:29.116377 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.116494 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.116645 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260924 10:44:29.116869 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.108674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:37843 (local address 127.2.182.62:39759)
0924 10:44:29.108799 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.108806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.109079 (+ 273us) server_negotiation.cc:407] Connection header received
0924 10:44:29.109245 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.109249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.109298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.109370 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.109771 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.110532 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.111854 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.112213 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.112332 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.113472 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.113479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.113485 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.113514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.115150 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.115637 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.115640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.115641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.115707 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.116036 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.116038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.116039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.116174 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.116241 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.116564 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.116689 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:29.136201 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.128272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.2.182.62:39759)
0924 10:44:29.128397 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.128400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.128411 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:44:29.128442 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.128445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.128479 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.128572 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.129329 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.129838 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.130658 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.130876 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.130990 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.133200 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.133206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.133213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.133242 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.134463 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.134904 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.134918 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.134924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.134998 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.135404 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.135408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.135409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.135576 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.135708 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.135928 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.136025 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:44:29.138682 3892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:29.141628 3892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:29.156690 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.149915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40831 (local address 127.0.0.1:37592)
0924 10:44:29.150704 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.150717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.150790 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.151418 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.151420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.151434 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:29.151515 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.151523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.152949 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.152952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.154460 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.154474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.156190 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.156194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.156302 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.156339 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.156438 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.156548 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":49,"thread_start_us":121,"threads_started":1}
I20260924 10:44:29.156829 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.149376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.2.182.2:43195)
0924 10:44:29.149687 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.149701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.150089 (+ 388us) server_negotiation.cc:407] Connection header received
0924 10:44:29.150377 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.150385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.150583 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.150731 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:29.151238 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.152517 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.155204 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.155963 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.156038 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.156193 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.156563 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.156657 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":112,"threads_started":1}
I20260924 10:44:29.157128 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.150253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.2.182.1:40831)
0924 10:44:29.150524 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.150537 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.150881 (+ 344us) server_negotiation.cc:407] Connection header received
0924 10:44:29.150954 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.150958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.151165 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.151290 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:29.151702 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.152797 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.154631 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.155866 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.155970 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.156127 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.156880 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.156988 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
I20260924 10:44:29.157346 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.149214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:43195 (local address 127.0.0.1:37640)
0924 10:44:29.149925 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.149994 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.150191 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.150918 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.150924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.150944 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:29.151062 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.151067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.152747 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.152763 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.155042 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.155051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.156295 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.156298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.156402 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.156435 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.157027 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.157130 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":441,"thread_start_us":67,"threads_started":1}
I20260924 10:44:29.157531 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.149386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44953 (local address 127.0.0.1:50118)
0924 10:44:29.150164 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.150181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.150269 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.151589 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.151592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.151607 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:29.151698 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.151703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.153103 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.153106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.154457 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.154467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.156543 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.156547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.156657 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.156677 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.157321 (+ 644us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.157410 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":37,"thread_start_us":93,"threads_started":1}
I20260924 10:44:29.157941 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.149869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.2.182.3:44953)
0924 10:44:29.150353 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.150365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.150386 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:44:29.150458 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.150463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.151454 (+ 991us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.151825 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:29.151959 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.152994 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.154596 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.156252 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.157503 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.157656 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.157702 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.157804 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":136,"threads_started":1}
I20260924 10:44:29.160616 4309 tablet_service.cc:1467] Processing CreateTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc (DEFAULT_TABLE table=test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.160722 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc (DEFAULT_TABLE table=test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.160722 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc (DEFAULT_TABLE table=test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.161594 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bd7f1242ac4eebb4a0087636342fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.161672 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bd7f1242ac4eebb4a0087636342fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.161672 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bd7f1242ac4eebb4a0087636342fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.168572 4398 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Bootstrap starting.
I20260924 10:44:29.170415 4398 tablet_bootstrap.cc:654] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.170794 4399 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Bootstrap starting.
I20260924 10:44:29.170832 4400 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Bootstrap starting.
I20260924 10:44:29.171269 4398 log.cc:824] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:29.172555 4399 tablet_bootstrap.cc:654] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.172763 4400 tablet_bootstrap.cc:654] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.173338 4398 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: No bootstrap required, opened a new log
I20260924 10:44:29.173400 4399 log.cc:824] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:29.173545 4398 ts_tablet_manager.cc:1397] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:44:29.173532 4400 log.cc:824] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:29.175053 4400 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: No bootstrap required, opened a new log
I20260924 10:44:29.175237 4400 ts_tablet_manager.cc:1397] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:44:29.175240 4399 tablet_bootstrap.cc:492] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: No bootstrap required, opened a new log
I20260924 10:44:29.175411 4399 ts_tablet_manager.cc:1397] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:44:29.177315 4398 raft_consensus.cc:348] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.177556 4398 raft_consensus.cc:374] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.177615 4398 raft_consensus.cc:729] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.178073 4400 raft_consensus.cc:348] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.178306 4400 raft_consensus.cc:374] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.178227 4398 consensus_queue.cc:260] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.178357 4400 raft_consensus.cc:729] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fc78eeb4244a2c93db2751a42c943d, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.178727 4400 consensus_queue.cc:260] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.178900 4375 heartbeater.cc:503] Master 127.2.182.62:39759 was elected leader, sending a full tablet report...
I20260924 10:44:29.179072 4399 raft_consensus.cc:348] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.179284 4235 heartbeater.cc:503] Master 127.2.182.62:39759 was elected leader, sending a full tablet report...
I20260924 10:44:29.179291 4399 raft_consensus.cc:374] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.179335 4399 raft_consensus.cc:729] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0ce87e1b8143c5a5c8bfc04ec71e39, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.179606 4400 ts_tablet_manager.cc:1428] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 10:44:29.179812 4399 consensus_queue.cc:260] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.180662 4399 ts_tablet_manager.cc:1428] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:44:29.181005 4095 heartbeater.cc:503] Master 127.2.182.62:39759 was elected leader, sending a full tablet report...
I20260924 10:44:29.181833 4398 ts_tablet_manager.cc:1428] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260924 10:44:29.353173 4376 tablet.cc:2369] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:44:29.408313 4236 tablet.cc:2369] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:44:29.425823 4096 tablet.cc:2369] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:29.492378 4404 raft_consensus.cc:482] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:29.492640 4404 raft_consensus.cc:504] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.493830 4404 leader_election.cc:288] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195), 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831)
I20260924 10:44:29.497517 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.494242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:60883 (local address 127.2.182.2:43195)
0924 10:44:29.494387 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.494393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.494415 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:44:29.494506 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.494510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.494565 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.494674 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:29.494961 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.495473 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.496505 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.496973 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.497090 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.497159 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.497334 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.497400 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:29.497517 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.494149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:43195 (local address 127.2.182.3:60883)
0924 10:44:29.494284 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.494301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.494401 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.494691 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.494695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.494715 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:29.494841 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.494847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.495590 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.495594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.496383 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.496391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.497093 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.497097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.497206 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.497235 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.497319 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.497389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:44:29.498162 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bd7f1242ac4eebb4a0087636342fbc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d" is_pre_election: true
I20260924 10:44:29.498222 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.494420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40831 (local address 127.2.182.3:46535)
0924 10:44:29.494831 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.494841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.494953 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.495221 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.495224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.495239 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:29.495624 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.495631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.496335 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.496341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.497104 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.497118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.497810 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.497814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.497920 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.497948 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.498036 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.498099 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":127,"threads_started":1}
I20260924 10:44:29.498242 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.494591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:46535 (local address 127.2.182.1:40831)
0924 10:44:29.494734 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.494738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.494920 (+ 182us) server_negotiation.cc:407] Connection header received
0924 10:44:29.495074 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.495077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.495121 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.495210 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:29.495744 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.496224 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.497236 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.497694 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.497795 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.497851 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.498056 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.498137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:44:29.498495 4189 raft_consensus.cc:2454] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.498857 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bd7f1242ac4eebb4a0087636342fbc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" is_pre_election: true
I20260924 10:44:29.499007 4261 leader_election.cc:302] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fc78eeb4244a2c93db2751a42c943d, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.499152 4049 raft_consensus.cc:2454] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.499282 4404 raft_consensus.cc:2781] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:29.499364 4404 raft_consensus.cc:482] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:29.499433 4404 raft_consensus.cc:3037] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.500460 4404 raft_consensus.cc:504] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.500777 4404 leader_election.cc:288] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Requested vote from peers 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195), 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831)
I20260924 10:44:29.501156 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bd7f1242ac4eebb4a0087636342fbc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d"
I20260924 10:44:29.501187 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bd7f1242ac4eebb4a0087636342fbc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
I20260924 10:44:29.501267 4189 raft_consensus.cc:3037] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.501307 4049 raft_consensus.cc:3037] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.502242 4189 raft_consensus.cc:2454] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.502393 4049 raft_consensus.cc:2454] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.502530 4261 leader_election.cc:302] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fc78eeb4244a2c93db2751a42c943d, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.502717 4404 raft_consensus.cc:2781] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:29.502925 4404 raft_consensus.cc:686] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Running, Role: LEADER
I20260924 10:44:29.503243 4404 consensus_queue.cc:237] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } }
I20260924 10:44:29.506474 3890 catalog_manager.cc:5495] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } health_report { overall_health: UNKNOWN } } }
I20260924 10:44:29.523113 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.519522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.2.182.3:44953)
0924 10:44:29.519656 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.519659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.519672 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:29.519839 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.519841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.519889 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.519969 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:29.520272 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.520745 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.521668 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.521867 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.521944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.522251 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:29.522634 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:29.522716 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.522865 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.522913 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:44:29.528663 4049 raft_consensus.cc:1261] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.528695 4189 raft_consensus.cc:1261] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.529379 4404 consensus_queue.cc:1035] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.529592 4408 consensus_queue.cc:1035] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.536513 4412 mvcc.cc:205] Tried to move back new op lower bound from 7332850358382317568 to 7332850358287609856. Current Snapshot: MvccSnapshot[applied={T|T < 7332850358382317568}]
I20260924 10:44:29.536716 4413 mvcc.cc:205] Tried to move back new op lower bound from 7332850358382317568 to 7332850358287609856. Current Snapshot: MvccSnapshot[applied={T|T < 7332850358382317568}]
I20260924 10:44:29.546038 3890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:29.546205 3890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:29.548241 3890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Sending DeleteTablet for 3 replicas of tablet 57bd7f1242ac4eebb4a0087636342fbc
I20260924 10:44:29.548991 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37640
I20260924 10:44:29.549046 4309 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:50118
I20260924 10:44:29.549093 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bd7f1242ac4eebb4a0087636342fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260924 10:44:29.549453 4423 tablet_replica.cc:330] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: stopping tablet replica
I20260924 10:44:29.549628 4421 tablet_replica.cc:330] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: stopping tablet replica
I20260924 10:44:29.549731 4423 raft_consensus.cc:2229] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:29.550032 4421 raft_consensus.cc:2229] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:29.550081 4423 raft_consensus.cc:2258] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.550037 3890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:44:29.550400 4422 tablet_replica.cc:330] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: stopping tablet replica
W20260924 10:44:29.550442 3890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:29.550632 4422 raft_consensus.cc:2229] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:29.550755 4421 raft_consensus.cc:2258] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.550881 4422 raft_consensus.cc:2258] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.552302 4422 ts_tablet_manager.cc:1905] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.552305 4423 ts_tablet_manager.cc:1905] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.553112 4421 ts_tablet_manager.cc:1905] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.554888 4422 ts_tablet_manager.cc:1918] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:29.554979 4422 log.cc:1196] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/57bd7f1242ac4eebb4a0087636342fbc
I20260924 10:44:29.555279 4422 ts_tablet_manager.cc:1939] T 57bd7f1242ac4eebb4a0087636342fbc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting consensus metadata
I20260924 10:44:29.555588 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 80ae393077c44deaa76085148df3f893 (DEFAULT_TABLE table=test-table [id=2b61ea33998c47c3a5b168d6195be08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.555832 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ae393077c44deaa76085148df3f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.557144 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 80ae393077c44deaa76085148df3f893 (DEFAULT_TABLE table=test-table [id=2b61ea33998c47c3a5b168d6195be08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.557379 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ae393077c44deaa76085148df3f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.557785 4421 ts_tablet_manager.cc:1918] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:29.557875 4421 log.cc:1196] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/57bd7f1242ac4eebb4a0087636342fbc
I20260924 10:44:29.558033 4423 ts_tablet_manager.cc:1918] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:29.558166 4423 log.cc:1196] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/57bd7f1242ac4eebb4a0087636342fbc
I20260924 10:44:29.558238 4421 ts_tablet_manager.cc:1939] T 57bd7f1242ac4eebb4a0087636342fbc P 49fc78eeb4244a2c93db2751a42c943d: Deleting consensus metadata
I20260924 10:44:29.558686 4423 ts_tablet_manager.cc:1939] T 57bd7f1242ac4eebb4a0087636342fbc P 9ea1cd291cf741d6adcdea2f1519d779: Deleting consensus metadata
I20260924 10:44:29.559593 4309 tablet_service.cc:1467] Processing CreateTablet for tablet 80ae393077c44deaa76085148df3f893 (DEFAULT_TABLE table=test-table [id=2b61ea33998c47c3a5b168d6195be08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:29.559756 4400 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Bootstrap starting.
I20260924 10:44:29.559834 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ae393077c44deaa76085148df3f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.559921 3876 catalog_manager.cc:4841] TS 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3:44953): tablet 57bd7f1242ac4eebb4a0087636342fbc (table test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]) successfully deleted
I20260924 10:44:29.560086 3877 catalog_manager.cc:4841] TS 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831): tablet 57bd7f1242ac4eebb4a0087636342fbc (table test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]) successfully deleted
I20260924 10:44:29.560226 3877 catalog_manager.cc:4841] TS 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195): tablet 57bd7f1242ac4eebb4a0087636342fbc (table test-table [id=eac09eb9cb7f405eb3f7fc7a5ce74ccd]) successfully deleted
I20260924 10:44:29.560735 4400 tablet_bootstrap.cc:654] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.561751 4399 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Bootstrap starting.
I20260924 10:44:29.562155 4398 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Bootstrap starting.
I20260924 10:44:29.562649 4399 tablet_bootstrap.cc:654] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.563010 4398 tablet_bootstrap.cc:654] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.563767 4400 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: No bootstrap required, opened a new log
I20260924 10:44:29.563849 4400 ts_tablet_manager.cc:1397] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260924 10:44:29.564056 4399 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: No bootstrap required, opened a new log
I20260924 10:44:29.564127 4399 ts_tablet_manager.cc:1397] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:29.564139 4398 tablet_bootstrap.cc:492] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: No bootstrap required, opened a new log
I20260924 10:44:29.564208 4398 ts_tablet_manager.cc:1397] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:29.564344 4400 raft_consensus.cc:348] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.564466 4400 raft_consensus.cc:374] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.564500 4400 raft_consensus.cc:729] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fc78eeb4244a2c93db2751a42c943d, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.564520 4399 raft_consensus.cc:348] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.564620 4399 raft_consensus.cc:374] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.564658 4399 raft_consensus.cc:729] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0ce87e1b8143c5a5c8bfc04ec71e39, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.564615 4398 raft_consensus.cc:348] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.564626 4400 consensus_queue.cc:260] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.564710 4398 raft_consensus.cc:374] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.564740 4398 raft_consensus.cc:729] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.564764 4399 consensus_queue.cc:260] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.564877 4400 ts_tablet_manager.cc:1428] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:29.564844 4398 consensus_queue.cc:260] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.565016 4399 ts_tablet_manager.cc:1428] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:29.565095 4398 ts_tablet_manager.cc:1428] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:29.696039 4417 raft_consensus.cc:482] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:29.696190 4417 raft_consensus.cc:504] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.696614 4417 leader_election.cc:288] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831), 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195)
I20260924 10:44:29.697141 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ae393077c44deaa76085148df3f893" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" is_pre_election: true
I20260924 10:44:29.697261 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ae393077c44deaa76085148df3f893" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d" is_pre_election: true
I20260924 10:44:29.697316 4049 raft_consensus.cc:2454] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.697417 4189 raft_consensus.cc:2454] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.697669 4261 leader_election.cc:302] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fc78eeb4244a2c93db2751a42c943d, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.697894 4417 raft_consensus.cc:2781] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:29.697976 4417 raft_consensus.cc:482] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:29.698025 4417 raft_consensus.cc:3037] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.698983 4417 raft_consensus.cc:504] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.699311 4417 leader_election.cc:288] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Requested vote from peers 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831), 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195)
I20260924 10:44:29.699656 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ae393077c44deaa76085148df3f893" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
I20260924 10:44:29.699689 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ae393077c44deaa76085148df3f893" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d"
I20260924 10:44:29.699790 4049 raft_consensus.cc:3037] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.699817 4189 raft_consensus.cc:3037] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.700706 4189 raft_consensus.cc:2454] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.700848 4049 raft_consensus.cc:2454] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.700997 4261 leader_election.cc:302] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fc78eeb4244a2c93db2751a42c943d, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.701179 4417 raft_consensus.cc:2781] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:29.701264 4417 raft_consensus.cc:686] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Running, Role: LEADER
I20260924 10:44:29.701447 4417 consensus_queue.cc:237] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.702996 3892 catalog_manager.cc:5495] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } health_report { overall_health: HEALTHY } } }
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246668, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:29.742048 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.734351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:39759 (local address 127.0.0.1:37556)
0924 10:44:29.734625 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.734639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.734725 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.734950 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.734953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.735310 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:29.735401 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.735407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.736102 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.736104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.736624 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.736630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.737081 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.737084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.737142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.737606 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:29.737617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:29.738882 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:29.740467 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.740477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.740484 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.740698 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.741015 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:29.741019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:29.741021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:29.741099 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:29.741483 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:29.741492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:29.741593 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.741709 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.741861 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260924 10:44:29.742048 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.734462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.2.182.62:39759)
0924 10:44:29.734704 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.734711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.734724 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:29.734805 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.734807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.734842 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.734915 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.735499 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.736013 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.736716 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.736994 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.737069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.739009 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.739012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.739014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.739031 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.740370 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.740785 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.740791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.740793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.740885 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.741187 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.741190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.741192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.741376 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.741475 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.741753 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.741916 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260924 10:44:29.745534 3892 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:44:29.746135 3892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:29.750298 4029 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa5682057c4d1f8be7d216498b23fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0e7ecb8adf4588a8fbfa056c5aecd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:29.750402 4169 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa5682057c4d1f8be7d216498b23fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0e7ecb8adf4588a8fbfa056c5aecd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:29.750554 4309 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa5682057c4d1f8be7d216498b23fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0e7ecb8adf4588a8fbfa056c5aecd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:29.750716 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa5682057c4d1f8be7d216498b23fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.750782 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa5682057c4d1f8be7d216498b23fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.750862 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa5682057c4d1f8be7d216498b23fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:29.753393 4400 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: Bootstrap starting.
I20260924 10:44:29.753468 4398 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: Bootstrap starting.
I20260924 10:44:29.754186 4398 tablet_bootstrap.cc:654] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.754245 4400 tablet_bootstrap.cc:654] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.754693 4399 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Bootstrap starting.
I20260924 10:44:29.755473 4400 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: No bootstrap required, opened a new log
I20260924 10:44:29.755561 4400 ts_tablet_manager.cc:1397] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:44:29.755575 4399 tablet_bootstrap.cc:654] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:29.756016 4400 raft_consensus.cc:348] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.756148 4400 raft_consensus.cc:374] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.756187 4400 raft_consensus.cc:729] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fc78eeb4244a2c93db2751a42c943d, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.756309 4400 consensus_queue.cc:260] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.756495 4398 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: No bootstrap required, opened a new log
I20260924 10:44:29.756561 4398 ts_tablet_manager.cc:1397] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 10:44:29.756562 4405 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.756697 4405 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.756733 4399 tablet_bootstrap.cc:492] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: No bootstrap required, opened a new log
I20260924 10:44:29.756803 4399 ts_tablet_manager.cc:1397] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:29.756875 4398 raft_consensus.cc:348] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.757036 4398 raft_consensus.cc:374] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.757066 4398 raft_consensus.cc:729] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.757170 4398 consensus_queue.cc:260] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.757217 4399 raft_consensus.cc:348] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.757340 4399 raft_consensus.cc:374] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:29.757390 4399 raft_consensus.cc:729] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0ce87e1b8143c5a5c8bfc04ec71e39, State: Initialized, Role: FOLLOWER
I20260924 10:44:29.757440 4398 ts_tablet_manager.cc:1428] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:29.757477 4417 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.757512 4399 consensus_queue.cc:260] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.757597 4417 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.757757 4399 ts_tablet_manager.cc:1428] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:29.757947 4406 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.758054 4406 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.756573 4400 ts_tablet_manager.cc:1428] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:29.834120 4417 raft_consensus.cc:482] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:29.834254 4417 raft_consensus.cc:504] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.834666 4417 leader_election.cc:288] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831), 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195)
I20260924 10:44:29.835228 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa5682057c4d1f8be7d216498b23fc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d" is_pre_election: true
I20260924 10:44:29.835230 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa5682057c4d1f8be7d216498b23fc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" is_pre_election: true
I20260924 10:44:29.835424 4189 raft_consensus.cc:2454] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.835425 4049 raft_consensus.cc:2454] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 0.
I20260924 10:44:29.835769 4261 leader_election.cc:302] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fc78eeb4244a2c93db2751a42c943d, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.836024 4417 raft_consensus.cc:2781] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:29.836091 4417 raft_consensus.cc:482] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:29.836130 4417 raft_consensus.cc:3037] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.837677 4417 raft_consensus.cc:504] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.838056 4417 leader_election.cc:288] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Requested vote from peers 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831), 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195)
I20260924 10:44:29.838374 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa5682057c4d1f8be7d216498b23fc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39"
I20260924 10:44:29.838502 4049 raft_consensus.cc:3037] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.838479 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa5682057c4d1f8be7d216498b23fc" candidate_uuid: "9ea1cd291cf741d6adcdea2f1519d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fc78eeb4244a2c93db2751a42c943d"
I20260924 10:44:29.838593 4189 raft_consensus.cc:3037] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:29.839602 4189 raft_consensus.cc:2454] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.839608 4049 raft_consensus.cc:2454] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea1cd291cf741d6adcdea2f1519d779 in term 1.
I20260924 10:44:29.839900 4261 leader_election.cc:302] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0ce87e1b8143c5a5c8bfc04ec71e39, 9ea1cd291cf741d6adcdea2f1519d779; no voters:
I20260924 10:44:29.840075 4417 raft_consensus.cc:2781] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:29.840340 4417 raft_consensus.cc:686] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea1cd291cf741d6adcdea2f1519d779, State: Running, Role: LEADER
I20260924 10:44:29.840507 4417 consensus_queue.cc:237] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } }
I20260924 10:44:29.841420 4408 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: TxnStatusTablet state changed. Reason: New leader 9ea1cd291cf741d6adcdea2f1519d779. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.841534 4408 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:29.841790 4439 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:44:29.841974 4439 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:44:29.842276 3890 catalog_manager.cc:5495] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } health_report { overall_health: HEALTHY } } }
I20260924 10:44:29.854377 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.850824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.2.182.3:44953)
0924 10:44:29.850938 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.850941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.851069 (+ 128us) server_negotiation.cc:407] Connection header received
0924 10:44:29.851243 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.851245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.851294 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.851356 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:29.851653 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.852114 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.853352 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.853556 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.853631 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.853937 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:29.854000 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:29.854074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.854194 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.854244 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:44:29.854410 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.850730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44953 (local address 127.0.0.1:50138)
0924 10:44:29.850964 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.850980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.851111 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.851403 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.851406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.851417 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:29.851532 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.851539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.852239 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.852243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.853236 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.853247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.853660 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.853664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.853758 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.853811 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:29.854083 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:29.854097 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.854175 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.854263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 10:44:29.856653 4049 raft_consensus.cc:1261] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.856716 4189 raft_consensus.cc:1261] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.857071 4417 consensus_queue.cc:1035] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.857191 4408 consensus_queue.cc:1035] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.858556 4405 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: TxnStatusTablet state changed. Reason: New leader 9ea1cd291cf741d6adcdea2f1519d779. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.858675 4405 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.858924 4406 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: TxnStatusTablet state changed. Reason: New leader 9ea1cd291cf741d6adcdea2f1519d779. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.859033 4406 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.860054 4408 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.860220 4408 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:29.860369 4411 mvcc.cc:205] Tried to move back new op lower bound from 7332850359728984064 to 7332850359668072448. Current Snapshot: MvccSnapshot[applied={T|T < 7332850359728984064}]
I20260924 10:44:29.861655 4404 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.861771 4404 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 9ea1cd291cf741d6adcdea2f1519d779: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:29.863270 4405 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.863390 4405 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 49fc78eeb4244a2c93db2751a42c943d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.863878 4406 tablet_replica.cc:439] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ea1cd291cf741d6adcdea2f1519d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 } } peers { permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 } } peers { permanent_uuid: "9ea1cd291cf741d6adcdea2f1519d779" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44953 } } }
I20260924 10:44:29.863986 4406 tablet_replica.cc:442] T e4fa5682057c4d1f8be7d216498b23fc P 7e0ce87e1b8143c5a5c8bfc04ec71e39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:29.875859 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.872769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44953 (local address 127.2.182.3:48825)
0924 10:44:29.873050 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:29.873061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:29.873175 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:29.873403 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:29.873406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:29.873412 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:29.873543 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.873549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.874227 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.874230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.874830 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:29.874837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.875230 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.875234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:29.875315 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.875357 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:29.875618 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:29.875625 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:44:29.875678 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 10:44:29.875750 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":31,"thread_start_us":132,"threads_started":1}
I20260924 10:44:29.875859 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.872839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:48825 (local address 127.2.182.3:44953)
0924 10:44:29.872973 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.872977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.873124 (+ 147us) server_negotiation.cc:407] Connection header received
0924 10:44:29.873273 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.873275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.873317 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.873397 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:29.873637 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.874122 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.874932 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.875133 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.875231 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.875485 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:29.875542 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:29.875608 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.875684 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.875728 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":35}
I20260924 10:44:29.881955 4049 raft_consensus.cc:1261] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.881968 4189 raft_consensus.cc:1261] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Refusing update from remote peer 9ea1cd291cf741d6adcdea2f1519d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:29.882413 4417 consensus_queue.cc:1035] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e0ce87e1b8143c5a5c8bfc04ec71e39" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.882542 4408 consensus_queue.cc:1035] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fc78eeb4244a2c93db2751a42c943d" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:29.926702 3889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:29.926862 3889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:29.928854 3889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20618ce53a444177a10091af89e60309: Sending DeleteTablet for 3 replicas of tablet 80ae393077c44deaa76085148df3f893
I20260924 10:44:29.929358 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ae393077c44deaa76085148df3f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260924 10:44:29.929421 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ae393077c44deaa76085148df3f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37640
I20260924 10:44:29.929528 4422 tablet_replica.cc:330] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: stopping tablet replica
I20260924 10:44:29.929579 4421 tablet_replica.cc:330] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: stopping tablet replica
I20260924 10:44:29.929626 4422 raft_consensus.cc:2229] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:29.929704 4421 raft_consensus.cc:2229] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:29.929733 4422 raft_consensus.cc:2258] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.929811 4421 raft_consensus.cc:2258] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.930250 4309 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ae393077c44deaa76085148df3f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:29 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:50118
I20260924 10:44:29.930397 4423 tablet_replica.cc:330] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: stopping tablet replica
I20260924 10:44:29.930455 4421 ts_tablet_manager.cc:1905] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.930488 4423 raft_consensus.cc:2229] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:29.930629 4423 raft_consensus.cc:2258] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:29.931249 4423 ts_tablet_manager.cc:1905] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.933192 4421 ts_tablet_manager.cc:1918] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:29.933254 4421 log.cc:1196] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/80ae393077c44deaa76085148df3f893
I20260924 10:44:29.933512 4421 ts_tablet_manager.cc:1939] T 80ae393077c44deaa76085148df3f893 P 49fc78eeb4244a2c93db2751a42c943d: Deleting consensus metadata
I20260924 10:44:29.934060 4423 ts_tablet_manager.cc:1918] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:29.934128 4423 log.cc:1196] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/80ae393077c44deaa76085148df3f893
I20260924 10:44:29.934108 3877 catalog_manager.cc:4841] TS 49fc78eeb4244a2c93db2751a42c943d (127.2.182.2:43195): tablet 80ae393077c44deaa76085148df3f893 (table test-table [id=2b61ea33998c47c3a5b168d6195be08e]) successfully deleted
I20260924 10:44:29.934377 4423 ts_tablet_manager.cc:1939] T 80ae393077c44deaa76085148df3f893 P 9ea1cd291cf741d6adcdea2f1519d779: Deleting consensus metadata
I20260924 10:44:29.934967 3876 catalog_manager.cc:4841] TS 9ea1cd291cf741d6adcdea2f1519d779 (127.2.182.3:44953): tablet 80ae393077c44deaa76085148df3f893 (table test-table [id=2b61ea33998c47c3a5b168d6195be08e]) successfully deleted
I20260924 10:44:29.935323 4422 ts_tablet_manager.cc:1905] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:29.937428 4422 ts_tablet_manager.cc:1918] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:29.937539 4422 log.cc:1196] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/80ae393077c44deaa76085148df3f893
I20260924 10:44:29.937805 4422 ts_tablet_manager.cc:1939] T 80ae393077c44deaa76085148df3f893 P 7e0ce87e1b8143c5a5c8bfc04ec71e39: Deleting consensus metadata
I20260924 10:44:29.938447 3877 catalog_manager.cc:4841] TS 7e0ce87e1b8143c5a5c8bfc04ec71e39 (127.2.182.1:40831): tablet 80ae393077c44deaa76085148df3f893 (table test-table [id=2b61ea33998c47c3a5b168d6195be08e]) successfully deleted
Sep 24 10:44:29 dist-test-slave-6lwt krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:29.947397 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.935331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.2.182.1:40831)
0924 10:44:29.935443 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.935448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.937239 (+ 1791us) server_negotiation.cc:407] Connection header received
0924 10:44:29.937360 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.937363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.937413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.937520 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.938480 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.939311 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.940102 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.940472 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.940615 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.944017 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.944025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.944036 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.944056 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.945736 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.946227 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.946232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.946233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.946310 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.946586 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.946588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.946589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.946793 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.946929 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.947161 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.947246 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 10:44:29.947818 4089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37608
I20260924 10:44:29.957490 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:29.949487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.2.182.62:39759)
0924 10:44:29.949598 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:29.949602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:29.949693 (+ 91us) server_negotiation.cc:407] Connection header received
0924 10:44:29.949824 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:29.949827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:29.949877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:29.949972 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:29.950681 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.951602 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.952321 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:29.952693 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:29.952800 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:29.954073 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:29.954077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:29.954080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:29.954116 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:29.955731 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.956284 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.956288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.956290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.956400 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:29.956679 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:29.956681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:29.956682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:29.956853 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:29.957032 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:29.957251 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:44:29.957356 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 10:44:29.957963 3889 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37568
I20260924 10:44:29.959268 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3964
I20260924 10:44:29.965610 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4104
I20260924 10:44:29.972250 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4244
I20260924 10:44:29.979104 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 3860
2026-09-24T10:44:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2911 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:30 dist-test-slave-6lwt krb5kdc[4457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:30 dist-test-slave-6lwt krb5kdc[4457](info): set up 2 sockets
Sep 24 10:44:30 dist-test-slave-6lwt krb5kdc[4457](info): commencing operation
krb5kdc: starting...
W20260924 10:44:30.980842 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:31.098161 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36075
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:31.205292 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:31.205525 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:31.205562 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:31.209298 4470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:31.209344 4470 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:44:31.209360 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:31.209378 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:31.209393 4470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:31.209406 4470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:31.214357 4470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36075
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:31.215250 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:31.216126 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:31.224349 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.224445 4470 server_base.cc:1034] running on GCE node
W20260924 10:44:31.224440 4478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:31.224349 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.225126 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:31.226117 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:31.227315 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790246671227303 us; error 24 us; skew 500 ppm
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:31.229998 4470 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:31.230741 4470 webserver.cc:460] Webserver started at http://127.2.182.62:39669/ using document root <none> and password file <none>
I20260924 10:44:31.231204 4470 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:31.231256 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:31.231412 4470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:31.233286 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "fed902b70be94dc780f34e1cf45f9408"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1a8ad05e20bc5fc630599d2fc951341a"
server_key_iv: "9e9dccca49f61e969fdf672d074e3861"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.233791 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "fed902b70be94dc780f34e1cf45f9408"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1a8ad05e20bc5fc630599d2fc951341a"
server_key_iv: "9e9dccca49f61e969fdf672d074e3861"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.237092 4470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:31.239336 4485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.240463 4470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:31.240602 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "fed902b70be94dc780f34e1cf45f9408"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1a8ad05e20bc5fc630599d2fc951341a"
server_key_iv: "9e9dccca49f61e969fdf672d074e3861"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.240717 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:31.253296 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:31.255870 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:31.256065 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:31.264099 4470 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36075
I20260924 10:44:31.264106 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36075 every 8 connection(s)
I20260924 10:44:31.265072 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:31.267633 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:31.272562 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Bootstrap starting.
I20260924 10:44:31.273388 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4470
I20260924 10:44:31.273475 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:31.273746 2776 external_mini_cluster.cc:1428] Setting key 30a0fa740a9675ec1a73b705e37b1e30
I20260924 10:44:31.275090 4548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:31.275774 4548 log.cc:824] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:31.277590 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: No bootstrap required, opened a new log
I20260924 10:44:31.280823 4548 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } }
I20260924 10:44:31.281056 4548 raft_consensus.cc:374] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:31.281121 4548 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed902b70be94dc780f34e1cf45f9408, State: Initialized, Role: FOLLOWER
I20260924 10:44:31.281586 4548 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } }
I20260924 10:44:31.281723 4548 raft_consensus.cc:388] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:31.281824 4548 raft_consensus.cc:482] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:31.281929 4548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:31.282739 4548 raft_consensus.cc:504] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } }
I20260924 10:44:31.283124 4548 leader_election.cc:302] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fed902b70be94dc780f34e1cf45f9408; no voters:
I20260924 10:44:31.283483 4548 leader_election.cc:288] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:31.283805 4553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:31.283970 4553 raft_consensus.cc:686] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [term 1 LEADER]: Becoming Leader. State: Replica: fed902b70be94dc780f34e1cf45f9408, State: Running, Role: LEADER
I20260924 10:44:31.284233 4553 consensus_queue.cc:237] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } }
I20260924 10:44:31.284672 4548 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:31.285324 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fed902b70be94dc780f34e1cf45f9408. Latest consensus state: current_term: 1 leader_uuid: "fed902b70be94dc780f34e1cf45f9408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } } }
I20260924 10:44:31.285482 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:31.286137 4554 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fed902b70be94dc780f34e1cf45f9408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed902b70be94dc780f34e1cf45f9408" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36075 } } }
I20260924 10:44:31.286244 4554 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:31.286948 4561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:31.288570 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.274759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.2.182.62:36075)
0924 10:44:31.275239 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.275254 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.275286 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:44:31.275964 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.275981 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.276399 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.276742 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.277523 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.278509 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.279330 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.279790 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.279953 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.282571 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.282578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.282589 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.282613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.284536 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.285050 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.285060 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.285064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.285168 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.285702 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.285705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.285707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.286176 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.287127 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.287401 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.287689 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":134,"threads_started":1}
I20260924 10:44:31.289844 4561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:31.295220 4561 catalog_manager.cc:1347] Generated new cluster ID: ffd53ffd4a654cc1b1559fd9ca085140
I20260924 10:44:31.295274 4561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:31.303524 4561 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:31.304672 4561 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:31.314420 4561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Generated new TSK 0
I20260924 10:44:31.315171 4561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:31.348731 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36075
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:31.454813 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:31.455068 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:31.455142 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:31.459215 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:31.459293 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:31.459364 4574 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:44:31.459408 4574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:31.459472 4574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:31.464650 4574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:31.465657 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:31.466616 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:31.475909 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.475993 4574 server_base.cc:1034] running on GCE node
W20260924 10:44:31.475925 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:31.475915 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.476387 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:31.477049 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:31.478259 4574 hybrid_clock.cc:648] HybridClock initialized: now 1790246671478231 us; error 37 us; skew 500 ppm
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:31.481189 4574 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:31.482025 4574 webserver.cc:460] Webserver started at http://127.2.182.1:42451/ using document root <none> and password file <none>
I20260924 10:44:31.482546 4574 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:31.482618 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:31.482786 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:31.484932 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "3a544f107df24e6f89217ebc18565a78"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "37ced4bd72e20e5a387ce87bee8bc714"
server_key_iv: "a63fb19e5e435877ea33051b86462c30"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.485602 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "3a544f107df24e6f89217ebc18565a78"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "37ced4bd72e20e5a387ce87bee8bc714"
server_key_iv: "a63fb19e5e435877ea33051b86462c30"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.489291 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:31.491585 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.492733 4574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:31.492861 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "3a544f107df24e6f89217ebc18565a78"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "37ced4bd72e20e5a387ce87bee8bc714"
server_key_iv: "a63fb19e5e435877ea33051b86462c30"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.493009 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:31.513451 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:31.516319 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:31.516531 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:31.518023 4574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:31.518077 4574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.518148 4574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:31.518186 4574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:31.530071 4574 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:46049
I20260924 10:44:31.530117 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:46049 every 8 connection(s)
I20260924 10:44:31.531117 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:31.532819 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.519191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.1:51671)
0924 10:44:31.519621 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.519667 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.520560 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.521250 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.521260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.521732 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.522240 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.522253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.523179 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.523184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.524365 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.524375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.525333 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.525338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.525564 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.526240 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.526266 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.528082 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.530203 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.530211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.530224 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.530477 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.530783 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.530785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.530786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.530900 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.531388 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.531392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.531662 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.531852 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.532138 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":137,"threads_started":1}
I20260924 10:44:31.533275 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.518863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:51671 (local address 127.2.182.62:36075)
0924 10:44:31.519052 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.519064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.519789 (+ 725us) server_negotiation.cc:407] Connection header received
0924 10:44:31.520811 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.520817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.520884 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.521024 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.522394 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.523058 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.524508 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.524977 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.525133 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.528283 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.528287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.528295 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.528318 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.530052 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.530581 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.530585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.530587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.530687 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.530990 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.530994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.530996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.531279 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.531379 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.533032 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.533156 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:44:31.534475 4705 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36075
I20260924 10:44:31.534637 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4574
I20260924 10:44:31.534685 4705 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:31.534772 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:31.535091 2776 external_mini_cluster.cc:1428] Setting key 1de4fe9758c824701256c251c4a1ed3e
I20260924 10:44:31.535243 4705 heartbeater.cc:511] Master 127.2.182.62:36075 requested a full tablet report, sending...
I20260924 10:44:31.536799 4501 ts_manager.cc:194] Registered new tserver with Master: 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049)
I20260924 10:44:31.538259 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:51671
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:31.545933 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.536991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:56629 (local address 127.2.182.62:36075)
0924 10:44:31.537084 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.537088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.539072 (+ 1984us) server_negotiation.cc:407] Connection header received
0924 10:44:31.539175 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.539177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.539213 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.539306 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.539742 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.540232 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.541175 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.541387 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.541492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.542524 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.542527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.542530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.542559 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.544191 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.544642 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.544645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.544648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.544735 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.545048 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.545051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.545052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.545280 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.545410 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.545681 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.545794 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 10:44:31.545946 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.1:56629)
0924 10:44:31.538975 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.538990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.539092 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.539322 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.539325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.539522 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.539633 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.539639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.540353 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.540356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.541054 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.541063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.541495 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.541498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.541594 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.542070 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.542081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.542391 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.544294 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.544298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.544300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.544548 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.544832 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.544837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.544839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.544943 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.545411 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.545414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.545503 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.545599 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.545751 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:31.561631 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36075
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:31.667646 4714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:31.667920 4714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:31.667997 4714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:31.671986 4714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:31.672065 4714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:31.672132 4714 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:44:31.672173 4714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:31.672232 4714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:31.677301 4714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:31.678249 4714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:31.679196 4714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:31.688371 4719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:31.688376 4720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.688508 4714 server_base.cc:1034] running on GCE node
W20260924 10:44:31.688416 4722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.689116 4714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:31.689733 4714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:31.690905 4714 hybrid_clock.cc:648] HybridClock initialized: now 1790246671690873 us; error 45 us; skew 500 ppm
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:31.693482 4714 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:31.694308 4714 webserver.cc:460] Webserver started at http://127.2.182.2:46803/ using document root <none> and password file <none>
I20260924 10:44:31.694841 4714 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:31.694926 4714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:31.695143 4714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:31.697228 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "e25e041baf0e4c5180d70dcdfebc0295"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "08f9fc8441195e3f0c2f9a3e8a5ea30c"
server_key_iv: "b26d40c9b55b5835dc9f5a2945fb0084"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.697788 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "e25e041baf0e4c5180d70dcdfebc0295"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "08f9fc8441195e3f0c2f9a3e8a5ea30c"
server_key_iv: "b26d40c9b55b5835dc9f5a2945fb0084"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.701270 4714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:31.703588 4729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.704746 4714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:31.704878 4714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "e25e041baf0e4c5180d70dcdfebc0295"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "08f9fc8441195e3f0c2f9a3e8a5ea30c"
server_key_iv: "b26d40c9b55b5835dc9f5a2945fb0084"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.705044 4714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:31.718446 4714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:31.721076 4714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:31.721256 4714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:31.722750 4714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:31.722842 4714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.722916 4714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:31.722944 4714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:31.733541 4714 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42387
I20260924 10:44:31.733865 4844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:42387 every 8 connection(s)
I20260924 10:44:31.734403 4714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:31.736750 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4714
I20260924 10:44:31.736864 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:31.736946 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.723427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:39759 (local address 127.2.182.62:36075)
0924 10:44:31.723557 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.723560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.725192 (+ 1632us) server_negotiation.cc:407] Connection header received
0924 10:44:31.726358 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.726362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.726424 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.726538 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.728182 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.729066 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.730222 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.730531 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.730641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.733197 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.733201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.733204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.733230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.734916 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.735477 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.735479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.735481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.735555 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.735860 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.735862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.735863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.736070 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.736146 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.736592 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.736714 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:44:31.737202 2776 external_mini_cluster.cc:1428] Setting key 22d3d6ae6b3374152605b014a0748926
I20260924 10:44:31.737682 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.724492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.2:39759)
0924 10:44:31.725052 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.725092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.726132 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.726720 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.726731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.727249 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.728000 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.728024 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.729180 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.729184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.730096 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.730105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.730676 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.730680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.730863 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.731469 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.731487 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.733044 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.735044 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.735051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.735063 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.735384 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.735626 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.735628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.735630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.735780 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.736158 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.736162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.736373 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.736521 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.736742 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":149,"threads_started":1}
I20260924 10:44:31.739399 4845 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36075
I20260924 10:44:31.739537 4845 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:31.740080 4845 heartbeater.cc:511] Master 127.2.182.62:36075 requested a full tablet report, sending...
I20260924 10:44:31.741178 4501 ts_manager.cc:194] Registered new tserver with Master: e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:31.741940 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:39759
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:44:31.748819 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.741531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:52307 (local address 127.2.182.62:36075)
0924 10:44:31.741658 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.741664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.741965 (+ 301us) server_negotiation.cc:407] Connection header received
0924 10:44:31.742161 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.742166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.742223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.742309 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.742741 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.743238 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.744134 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.744483 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.744598 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.745706 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.745709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.745712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.745734 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.747135 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.747623 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.747627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.747629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.747708 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.747986 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.747990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.747992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.748190 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.748305 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.748558 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.748665 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:44:31.748833 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.741449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.2:52307)
0924 10:44:31.741869 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.741883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.741984 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.742321 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.742324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.742534 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.742628 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.742634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.743375 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.743378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.744020 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.744029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.744601 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.744604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.744723 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.745263 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.745275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.745571 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.747257 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.747260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.747262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.747526 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.747806 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.747811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.747813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.747897 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.748293 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.748296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.748391 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.748497 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.748637 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:31.763567 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36075
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:31.870119 4854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:31.870352 4854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:31.870396 4854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:31.874401 4854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:31.874447 4854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:31.874513 4854 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:44:31.874545 4854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:31.874612 4854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:31.879557 4854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:31.880494 4854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:31.881467 4854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:31.891189 4862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:31.891198 4859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:31.891229 4860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:31.891466 4854 server_base.cc:1034] running on GCE node
I20260924 10:44:31.891891 4854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:31.892472 4854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:31.893637 4854 hybrid_clock.cc:648] HybridClock initialized: now 1790246671893614 us; error 23 us; skew 500 ppm
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:31.896210 4854 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:31.897112 4854 webserver.cc:460] Webserver started at http://127.2.182.3:46017/ using document root <none> and password file <none>
I20260924 10:44:31.897666 4854 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:31.897749 4854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:31.897972 4854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:31.899899 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "83e3116a3afa4ffca6264d82a1022d43"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1c34423f96ab8e71d0359ae6a494337a"
server_key_iv: "e343c3b992f260a0816ef6e81f8b2e9c"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.900424 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "83e3116a3afa4ffca6264d82a1022d43"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1c34423f96ab8e71d0359ae6a494337a"
server_key_iv: "e343c3b992f260a0816ef6e81f8b2e9c"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.903875 4854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 10:44:31.906227 4869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.907322 4854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:31.907444 4854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "83e3116a3afa4ffca6264d82a1022d43"
format_stamp: "Formatted at 2026-09-24 10:44:31 on dist-test-slave-6lwt"
server_key: "1c34423f96ab8e71d0359ae6a494337a"
server_key_iv: "e343c3b992f260a0816ef6e81f8b2e9c"
server_key_version: "encryptionkey@0"
I20260924 10:44:31.907541 4854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:31.925899 4854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:31.928613 4854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:31.928818 4854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:31.930126 4854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:31.930199 4854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:31.930266 4854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:31.930315 4854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:31.941041 4854 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:37393
I20260924 10:44:31.941167 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:37393 every 8 connection(s)
I20260924 10:44:31.941836 4854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:31.943315 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.931043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:43449 (local address 127.2.182.62:36075)
0924 10:44:31.931166 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.931170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.932163 (+ 993us) server_negotiation.cc:407] Connection header received
0924 10:44:31.933307 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.933310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.933369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.933462 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.935000 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.935754 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.936992 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.937221 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.937328 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.939918 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.939922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.939924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.939948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.941462 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.941899 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.941902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.941904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.941984 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.942327 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.942330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.942332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.942545 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.942628 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.943101 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.943193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:31.944027 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.931567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.3:43449)
0924 10:44:31.932019 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.932057 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.933023 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.933687 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.933698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.934158 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.934838 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.934862 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.935857 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.935861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.936838 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.936847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.937371 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.937375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.937527 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.938047 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.938072 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.939763 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.941560 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.941566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.941578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.941803 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.942065 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.942068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.942070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.942219 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.942678 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.942683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.942893 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.943035 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.943249 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260924 10:44:31.945303 4985 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36075
I20260924 10:44:31.945446 4985 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:31.945930 4985 heartbeater.cc:511] Master 127.2.182.62:36075 requested a full tablet report, sending...
I20260924 10:44:31.946897 4502 ts_manager.cc:194] Registered new tserver with Master: 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393)
I20260924 10:44:31.947569 4502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:43449
I20260924 10:44:31.949220 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 4854
I20260924 10:44:31.949324 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:31.949607 2776 external_mini_cluster.cc:1428] Setting key 361e6815bc81a45bfa1fb0cc8ebe1950
I20260924 10:44:31.951574 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:44:31.954114 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.947409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56147 (local address 127.2.182.62:36075)
0924 10:44:31.947529 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.947532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.947543 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:44:31.947641 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.947644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.947703 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.947766 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.948183 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.948866 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.949771 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.950095 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.950200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.951232 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.951235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.951238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.951259 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.952563 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.952985 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.952988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.952990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.953069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.953348 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.953351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.953352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.953508 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.953625 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.953877 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.953977 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:44:31.954133 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.947204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.2.182.3:56147)
0924 10:44:31.947454 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.947479 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.947554 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.947808 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.947812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.947991 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:31.948074 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.948079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.948995 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.948997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.949637 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.949661 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.950202 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.950206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.950326 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.950809 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:31.950820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:31.951107 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:31.952669 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.952674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.952676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.952862 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.953156 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:31.953160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:31.953162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:31.953244 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:31.953626 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:31.953630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:31.953720 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.953826 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.953960 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:44:31 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:31.975885 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.968553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.2.182.62:36075)
0924 10:44:31.968681 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.968685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.968697 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:31.968731 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.968733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.968766 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.968836 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:31.969554 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.970053 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.970841 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.971036 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.971106 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.973304 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:31.973307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:31.973310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:31.973329 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:31.974477 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.974847 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.974849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.974850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.974916 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:31.975127 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:31.975128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:31.975129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:31.975271 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:31.975363 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.975685 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.975767 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:31.979012 4502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:31.981873 4502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:31.995733 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.989846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42387 (local address 127.0.0.1:39896)
0924 10:44:31.990461 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.990509 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.990634 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.991886 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.991889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.991910 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:31.992013 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.992019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.993409 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.993412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.994433 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.994440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.995208 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.995212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.995320 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.995360 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.995469 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.995588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":68,"thread_start_us":78,"threads_started":1}
I20260924 10:44:31.995769 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.989977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.2.182.2:42387)
0924 10:44:31.991280 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.991297 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.991316 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:44:31.991386 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.991391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.991579 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.991751 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:31.992134 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.993273 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.994540 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.995088 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.995210 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.995371 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.995495 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.995632 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":139,"threads_started":1}
I20260924 10:44:31.996101 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.990626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.2.182.3:37393)
0924 10:44:31.991225 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.991239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.991261 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:44:31.991346 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.991353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.991522 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.991678 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:31.991962 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.992783 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.994329 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.995112 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.995208 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.995370 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.995809 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.995912 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":102,"threads_started":1}
I20260924 10:44:31.996178 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.989926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.0.0.1:53960)
0924 10:44:31.990836 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.990848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.990927 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.991665 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.991671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.991692 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:31.991841 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.991848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.992928 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.992934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.994201 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.994212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.995428 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.995431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.995525 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.995547 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.995614 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.995677 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 10:44:31.996769 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.990187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:46049 (local address 127.0.0.1:44042)
0924 10:44:31.991015 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:31.991028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:31.991277 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:31.991827 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:31.991829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:31.991844 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:31.991931 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.991938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.993409 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.993412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.995088 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:31.995099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.996416 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.996420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:31.996512 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.996533 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:44:31.996601 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:44:31.996661 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":594,"thread_start_us":95,"threads_started":1}
I20260924 10:44:31.998837 4919 tablet_service.cc:1467] Processing CreateTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 (DEFAULT_TABLE table=test-table [id=e96c566a0a0b4220b8512bf38baf69b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:31.998840 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 (DEFAULT_TABLE table=test-table [id=e96c566a0a0b4220b8512bf38baf69b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:31.999748 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a605cfd2a04080a59d1e3d88c2d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:31.999750 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a605cfd2a04080a59d1e3d88c2d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.001019 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:31.990626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.2.182.1:46049)
0924 10:44:31.991012 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:31.991025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:31.991265 (+ 240us) server_negotiation.cc:407] Connection header received
0924 10:44:31.991386 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:31.991391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:31.991579 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:31.991731 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:31.992057 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.993272 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.995230 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:31.996016 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:31.996121 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:31.996353 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:31.996694 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 10:44:31.996791 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260924 10:44:32.003667 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 (DEFAULT_TABLE table=test-table [id=e96c566a0a0b4220b8512bf38baf69b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:32.004637 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a605cfd2a04080a59d1e3d88c2d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.006516 5008 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Bootstrap starting.
I20260924 10:44:32.007519 5009 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Bootstrap starting.
I20260924 10:44:32.008625 5008 tablet_bootstrap.cc:654] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.009444 5008 log.cc:824] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:32.009656 5009 tablet_bootstrap.cc:654] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.010403 5009 log.cc:824] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:32.011338 5008 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: No bootstrap required, opened a new log
I20260924 10:44:32.011516 5008 ts_tablet_manager.cc:1397] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260924 10:44:32.012538 5009 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: No bootstrap required, opened a new log
I20260924 10:44:32.012702 5009 ts_tablet_manager.cc:1397] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:44:32.013191 5012 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Bootstrap starting.
I20260924 10:44:32.015184 5012 tablet_bootstrap.cc:654] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.015291 5008 raft_consensus.cc:348] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.015563 5008 raft_consensus.cc:374] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.015620 5008 raft_consensus.cc:729] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.015450 5009 raft_consensus.cc:348] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.015682 5009 raft_consensus.cc:374] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.015769 5009 raft_consensus.cc:729] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.015960 5012 log.cc:824] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:32.016048 5008 consensus_queue.cc:260] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.016199 5009 consensus_queue.cc:260] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.016848 4845 heartbeater.cc:503] Master 127.2.182.62:36075 was elected leader, sending a full tablet report...
I20260924 10:44:32.017012 5008 ts_tablet_manager.cc:1428] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:44:32.017138 5009 ts_tablet_manager.cc:1428] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260924 10:44:32.017303 4985 heartbeater.cc:503] Master 127.2.182.62:36075 was elected leader, sending a full tablet report...
I20260924 10:44:32.017714 5012 tablet_bootstrap.cc:492] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: No bootstrap required, opened a new log
I20260924 10:44:32.017831 5012 ts_tablet_manager.cc:1397] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:44:32.020807 5012 raft_consensus.cc:348] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.021064 5012 raft_consensus.cc:374] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.021132 5012 raft_consensus.cc:729] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.021584 5012 consensus_queue.cc:260] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.022135 4705 heartbeater.cc:503] Master 127.2.182.62:36075 was elected leader, sending a full tablet report...
I20260924 10:44:32.022455 5012 ts_tablet_manager.cc:1428] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:44:32.030753 5016 raft_consensus.cc:482] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:32.030903 5016 raft_consensus.cc:504] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.031792 5016 leader_election.cc:288] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387), 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393)
W20260924 10:44:32.032341 4706 tablet.cc:2369] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:32.035578 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.032080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:42691 (local address 127.2.182.2:42387)
0924 10:44:32.032186 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.032190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.032203 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:32.032381 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.032384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.032471 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.032546 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:32.032823 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.033592 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.034598 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.035058 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.035169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.035228 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.035406 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.035470 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:44:32.035599 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.031976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42387 (local address 127.2.182.1:42691)
0924 10:44:32.032137 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.032151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.032259 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.032558 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.032560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.032579 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:32.032690 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.032696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.033711 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.033713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.034456 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.034464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.035167 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.035171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.035289 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.035306 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.035390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.035455 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 10:44:32.035759 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.032088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:36615 (local address 127.2.182.3:37393)
0924 10:44:32.032205 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.032209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.032509 (+ 300us) server_negotiation.cc:407] Connection header received
0924 10:44:32.032682 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.032685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.032729 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.032824 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:32.033081 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.033780 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.034823 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.035233 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.035352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.035405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.035583 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.035645 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:44:32.035761 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.032017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.2.182.1:36615)
0924 10:44:32.032396 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.032420 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.032566 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.032836 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.032839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.032855 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:32.032980 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.032984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.033880 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.033886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.034698 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.034706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.035353 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.035357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.035454 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.035474 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.035568 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.035632 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":116,"threads_started":1}
I20260924 10:44:32.036238 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a605cfd2a04080a59d1e3d88c2d033" candidate_uuid: "3a544f107df24e6f89217ebc18565a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3116a3afa4ffca6264d82a1022d43" is_pre_election: true
I20260924 10:44:32.036353 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a605cfd2a04080a59d1e3d88c2d033" candidate_uuid: "3a544f107df24e6f89217ebc18565a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295" is_pre_election: true
I20260924 10:44:32.036482 4939 raft_consensus.cc:2454] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a544f107df24e6f89217ebc18565a78 in term 0.
I20260924 10:44:32.036616 4799 raft_consensus.cc:2454] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a544f107df24e6f89217ebc18565a78 in term 0.
I20260924 10:44:32.036972 4590 leader_election.cc:302] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, 83e3116a3afa4ffca6264d82a1022d43; no voters:
I20260924 10:44:32.037216 5016 raft_consensus.cc:2781] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:32.037295 5016 raft_consensus.cc:482] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:32.037362 5016 raft_consensus.cc:3037] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.038452 5016 raft_consensus.cc:504] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.038829 5016 leader_election.cc:288] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [CANDIDATE]: Term 1 election: Requested vote from peers e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387), 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393)
I20260924 10:44:32.039108 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a605cfd2a04080a59d1e3d88c2d033" candidate_uuid: "3a544f107df24e6f89217ebc18565a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295"
I20260924 10:44:32.039134 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a605cfd2a04080a59d1e3d88c2d033" candidate_uuid: "3a544f107df24e6f89217ebc18565a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3116a3afa4ffca6264d82a1022d43"
I20260924 10:44:32.039225 4939 raft_consensus.cc:3037] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.039242 4799 raft_consensus.cc:3037] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.040447 4939 raft_consensus.cc:2454] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a544f107df24e6f89217ebc18565a78 in term 1.
I20260924 10:44:32.040447 4799 raft_consensus.cc:2454] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a544f107df24e6f89217ebc18565a78 in term 1.
I20260924 10:44:32.041173 4591 leader_election.cc:302] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:32.041359 5016 raft_consensus.cc:2781] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:32.041541 5016 raft_consensus.cc:686] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 1 LEADER]: Becoming Leader. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Running, Role: LEADER
I20260924 10:44:32.041826 5016 consensus_queue.cc:237] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } }
I20260924 10:44:32.044611 4502 catalog_manager.cc:5495] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a544f107df24e6f89217ebc18565a78 (127.2.182.1). New cstate: current_term: 1 leader_uuid: "3a544f107df24e6f89217ebc18565a78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } health_report { overall_health: HEALTHY } } }
Sep 24 10:44:32 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:32.086742 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.078576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.2.182.62:36075)
0924 10:44:32.078717 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.078723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.078919 (+ 196us) server_negotiation.cc:407] Connection header received
0924 10:44:32.079071 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.079074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.079116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.079199 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:32.079834 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.080373 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.081170 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.081379 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.081485 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.083575 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:32.083578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:32.083581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:32.083608 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:32.084940 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:32.085484 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:32.085488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:32.085490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:32.085565 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:32.085829 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:32.085835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:32.085836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:32.086020 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:32.086125 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.086443 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.086596 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":50}
I20260924 10:44:32.086750 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.078532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36075 (local address 127.0.0.1:47710)
0924 10:44:32.078828 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.078839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.078975 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.079216 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.079219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.079621 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:32.079705 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.079711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.080484 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.080489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.081056 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.081064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.081486 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.081490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.081570 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.082089 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:32.082100 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:32.083451 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:32.085040 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:32.085051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:32.085057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:32.085363 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:32.085656 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:32.085659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:32.085661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:32.085743 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:32.086122 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:32.086129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:32.086262 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.086398 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.086545 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260924 10:44:32.090842 4502 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:47710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:44:32.091554 4502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:32.096113 4639 tablet_service.cc:1467] Processing CreateTablet for tablet dc9221ea01cb43f2bd2606470ae7bf07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18e70e32bfa54c9cb8a191988a453559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:32.096426 4779 tablet_service.cc:1467] Processing CreateTablet for tablet dc9221ea01cb43f2bd2606470ae7bf07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18e70e32bfa54c9cb8a191988a453559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:32.096920 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9221ea01cb43f2bd2606470ae7bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.096971 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9221ea01cb43f2bd2606470ae7bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.096746 4919 tablet_service.cc:1467] Processing CreateTablet for tablet dc9221ea01cb43f2bd2606470ae7bf07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18e70e32bfa54c9cb8a191988a453559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:44:32.097409 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9221ea01cb43f2bd2606470ae7bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.099658 5008 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: Bootstrap starting.
I20260924 10:44:32.099745 5009 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: Bootstrap starting.
I20260924 10:44:32.100535 5008 tablet_bootstrap.cc:654] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.100575 5009 tablet_bootstrap.cc:654] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.101181 5012 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: Bootstrap starting.
I20260924 10:44:32.101809 5008 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: No bootstrap required, opened a new log
I20260924 10:44:32.101900 5008 ts_tablet_manager.cc:1397] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:32.102017 5009 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: No bootstrap required, opened a new log
I20260924 10:44:32.102115 5009 ts_tablet_manager.cc:1397] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:32.102154 5012 tablet_bootstrap.cc:654] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.102380 5008 raft_consensus.cc:348] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.102501 5008 raft_consensus.cc:374] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.102545 5008 raft_consensus.cc:729] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.102586 5009 raft_consensus.cc:348] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.102716 5009 raft_consensus.cc:374] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.102669 5008 consensus_queue.cc:260] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.102756 5009 raft_consensus.cc:729] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.102907 5009 consensus_queue.cc:260] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.102996 5008 ts_tablet_manager.cc:1428] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:32.103163 5009 ts_tablet_manager.cc:1428] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:44:32.103159 5015 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.103351 5015 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.102965 5014 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.103689 5014 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.104038 5012 tablet_bootstrap.cc:492] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: No bootstrap required, opened a new log
I20260924 10:44:32.104123 5012 ts_tablet_manager.cc:1397] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:44:32.104562 5012 raft_consensus.cc:348] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.104701 5012 raft_consensus.cc:374] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.104739 5012 raft_consensus.cc:729] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.104871 5012 consensus_queue.cc:260] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.105158 5016 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.105309 5016 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.105168 5012 ts_tablet_manager.cc:1428] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:32.117897 5014 raft_consensus.cc:482] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:32.117998 5014 raft_consensus.cc:504] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.118988 5014 leader_election.cc:288] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:32.123466 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.119334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:45705 (local address 127.2.182.1:46049)
0924 10:44:32.119447 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.119451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.119464 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:32.119507 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.119509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.119553 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.119612 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:32.119922 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.120682 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122250 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.122865 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122976 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.123034 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.123277 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.123351 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:32.123467 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.119175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:46049 (local address 127.2.182.3:45705)
0924 10:44:32.119318 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.119334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.119414 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.119658 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.119661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.119673 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:32.119801 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.119805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.120782 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.120784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.122128 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.122138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.122989 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.123151 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.123179 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.123261 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.123346 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":91}
I20260924 10:44:32.123488 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.119258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:48323 (local address 127.2.182.2:42387)
0924 10:44:32.119379 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.119383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.119667 (+ 284us) server_negotiation.cc:407] Connection header received
0924 10:44:32.119745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.119747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.119785 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.119838 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:32.120079 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.120753 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122250 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.122865 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122976 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.123034 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.123327 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.123393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:32.123965 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.119175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42387 (local address 127.2.182.3:48323)
0924 10:44:32.119561 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.119571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.119670 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.119869 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.119874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.119895 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:32.119992 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.119996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.120853 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.120856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.122128 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.122138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.122996 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.122998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.123191 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.123216 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.123767 (+ 551us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.123837 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":190,"thread_start_us":111,"threads_started":1}
I20260924 10:44:32.124097 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9221ea01cb43f2bd2606470ae7bf07" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78" is_pre_election: true
I20260924 10:44:32.124315 4659 raft_consensus.cc:2454] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:32.124501 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9221ea01cb43f2bd2606470ae7bf07" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295" is_pre_election: true
I20260924 10:44:32.124632 4799 raft_consensus.cc:2454] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:32.124809 4872 leader_election.cc:302] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, 83e3116a3afa4ffca6264d82a1022d43; no voters:
I20260924 10:44:32.125079 5014 raft_consensus.cc:2781] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:32.125181 5014 raft_consensus.cc:482] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:32.125244 5014 raft_consensus.cc:3037] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.126219 5014 raft_consensus.cc:504] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.126577 5014 leader_election.cc:288] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Requested vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:32.126920 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9221ea01cb43f2bd2606470ae7bf07" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78"
I20260924 10:44:32.126920 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9221ea01cb43f2bd2606470ae7bf07" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295"
I20260924 10:44:32.127048 4659 raft_consensus.cc:3037] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.127048 4799 raft_consensus.cc:3037] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.128141 4659 raft_consensus.cc:2454] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:32.128146 4799 raft_consensus.cc:2454] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:32.128418 4872 leader_election.cc:302] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, 83e3116a3afa4ffca6264d82a1022d43; no voters:
I20260924 10:44:32.128602 5014 raft_consensus.cc:2781] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:32.128823 5014 raft_consensus.cc:686] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 LEADER]: Becoming Leader. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Running, Role: LEADER
I20260924 10:44:32.129177 5014 consensus_queue.cc:237] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.131412 5030 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: TxnStatusTablet state changed. Reason: New leader 83e3116a3afa4ffca6264d82a1022d43. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.131508 5030 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:32.131747 5032 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:44:32.131953 5032 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:44:32.132208 4500 catalog_manager.cc:5495] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } health_report { overall_health: HEALTHY } } }
I20260924 10:44:32.148947 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.144713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.0.0.1:53962)
0924 10:44:32.145044 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.145061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.145208 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.145578 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.145580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.145587 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:32.145676 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.145682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.146693 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.146696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.147306 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.147315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.147840 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.147843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.147920 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.147983 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.148652 (+ 669us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.148667 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.148727 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.148792 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260924 10:44:32.148967 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.144845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.2.182.3:37393)
0924 10:44:32.145069 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.145074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.145137 (+ 63us) server_negotiation.cc:407] Connection header received
0924 10:44:32.145387 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.145391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.145446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.145534 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.145823 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.146595 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.147412 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.147731 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.147838 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.148110 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.148544 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.148646 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.148747 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.148803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:44:32.153326 4799 raft_consensus.cc:1261] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.153321 4659 raft_consensus.cc:1261] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.153962 5014 consensus_queue.cc:1035] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:32.154088 5030 consensus_queue.cc:1035] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:32.158243 5015 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: TxnStatusTablet state changed. Reason: New leader 83e3116a3afa4ffca6264d82a1022d43. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.158349 5015 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.158300 5016 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: TxnStatusTablet state changed. Reason: New leader 83e3116a3afa4ffca6264d82a1022d43. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.158401 5016 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.160646 5014 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.160749 5014 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:32.161005 5015 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.161100 5015 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P e25e041baf0e4c5180d70dcdfebc0295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.166102 5030 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.166221 5030 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 83e3116a3afa4ffca6264d82a1022d43: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:44:32.166831 5038 mvcc.cc:205] Tried to move back new op lower bound from 7332850369133735936 to 7332850369043755008. Current Snapshot: MvccSnapshot[applied={T|T < 7332850369133735936}]
I20260924 10:44:32.167912 5016 tablet_replica.cc:439] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } } }
I20260924 10:44:32.168008 5016 tablet_replica.cc:442] T dc9221ea01cb43f2bd2606470ae7bf07 P 3a544f107df24e6f89217ebc18565a78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:44:32.168931 5036 mvcc.cc:205] Tried to move back new op lower bound from 7332850369133735936 to 7332850369043755008. Current Snapshot: MvccSnapshot[applied={T|T < 7332850369133735936}]
I20260924 10:44:32.170821 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.166527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.2.182.1:46049)
0924 10:44:32.166639 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.166653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.166665 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:32.166745 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.166748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.166808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.166901 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.167415 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.168131 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.169252 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.169563 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.169678 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.170088 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.170458 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.170579 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.170653 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.170701 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:32.180594 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.177500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:50751 (local address 127.2.182.3:37393)
0924 10:44:32.177607 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.177611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.177757 (+ 146us) server_negotiation.cc:407] Connection header received
0924 10:44:32.177877 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.177878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.177924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.178003 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.178271 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.178761 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.179588 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.179800 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.179903 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.180206 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.180269 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.180345 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.180432 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.180474 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:44:32.180635 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.177356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.2.182.1:50751)
0924 10:44:32.177668 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.177683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.177792 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.178017 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.178020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.178028 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:32.178162 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.178167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.178868 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.178873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.179476 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.179484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.179922 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.179925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.180020 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.180057 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.180353 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.180362 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.180424 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.180492 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":121,"threads_started":1}
I20260924 10:44:32.187016 4799 raft_consensus.cc:1261] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Refusing update from remote peer 3a544f107df24e6f89217ebc18565a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.187373 4939 raft_consensus.cc:1261] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Refusing update from remote peer 3a544f107df24e6f89217ebc18565a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.187462 5016 consensus_queue.cc:1035] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:32.187845 5016 consensus_queue.cc:1035] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:32.192734 5035 mvcc.cc:205] Tried to move back new op lower bound from 7332850369273151488 to 7332850368684531712. Current Snapshot: MvccSnapshot[applied={T|T < 7332850369273151488}]
I20260924 10:44:32.209234 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.205788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:46049 (local address 127.2.182.3:50239)
0924 10:44:32.205996 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.206010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.206084 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.206429 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.206431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.206438 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:32.206514 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.206518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.207449 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.207452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.207985 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.207992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.208525 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.208528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.208616 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.208652 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.208936 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.208942 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.209045 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.209131 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 10:44:32.209220 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.205842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:50239 (local address 127.2.182.1:46049)
0924 10:44:32.206221 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.206226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.206239 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:32.206278 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.206280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.206320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.206388 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.206608 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.207330 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.208117 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.208437 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.208528 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.208770 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.208841 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.208939 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.209059 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.209105 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260924 10:44:32.229816 4501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:32.229995 4501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:32.232352 4501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Sending DeleteTablet for 3 replicas of tablet 15a605cfd2a04080a59d1e3d88c2d033
I20260924 10:44:32.232964 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:32 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:39896
I20260924 10:44:32.233100 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:32 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44042
I20260924 10:44:32.233239 4919 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a605cfd2a04080a59d1e3d88c2d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:32 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:53960
I20260924 10:44:32.233551 5054 tablet_replica.cc:330] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: stopping tablet replica
I20260924 10:44:32.233919 5054 raft_consensus.cc:2229] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:32.233906 4501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:44:32.234311 5054 raft_consensus.cc:2258] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:44:32.234311 4501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:32.236514 5054 ts_tablet_manager.cc:1905] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:32.238889 5055 tablet_replica.cc:330] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: stopping tablet replica
I20260924 10:44:32.239217 5055 raft_consensus.cc:2229] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:32.239311 4639 tablet_service.cc:1467] Processing CreateTablet for tablet d6114e9d213e49fdacdb20352449c504 (DEFAULT_TABLE table=test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:32.239522 5055 raft_consensus.cc:2258] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:32.239562 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6114e9d213e49fdacdb20352449c504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.240091 5054 ts_tablet_manager.cc:1918] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:32.240270 5054 log.cc:1196] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/15a605cfd2a04080a59d1e3d88c2d033
I20260924 10:44:32.241318 5055 ts_tablet_manager.cc:1905] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:32.241712 5054 ts_tablet_manager.cc:1939] T 15a605cfd2a04080a59d1e3d88c2d033 P e25e041baf0e4c5180d70dcdfebc0295: Deleting consensus metadata
I20260924 10:44:32.242129 4919 tablet_service.cc:1467] Processing CreateTablet for tablet d6114e9d213e49fdacdb20352449c504 (DEFAULT_TABLE table=test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:32.242374 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6114e9d213e49fdacdb20352449c504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.244871 5012 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Bootstrap starting.
I20260924 10:44:32.245275 5008 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Bootstrap starting.
I20260924 10:44:32.245301 5056 tablet_replica.cc:330] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: stopping tablet replica
I20260924 10:44:32.245536 5056 raft_consensus.cc:2229] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:32.245698 5056 raft_consensus.cc:2258] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:32.246048 5055 ts_tablet_manager.cc:1918] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:32.246091 5008 tablet_bootstrap.cc:654] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.246197 5055 log.cc:1196] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/15a605cfd2a04080a59d1e3d88c2d033
I20260924 10:44:32.246554 5055 ts_tablet_manager.cc:1939] T 15a605cfd2a04080a59d1e3d88c2d033 P 3a544f107df24e6f89217ebc18565a78: Deleting consensus metadata
I20260924 10:44:32.246963 5056 ts_tablet_manager.cc:1905] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:32.247634 5012 tablet_bootstrap.cc:654] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.248281 4488 catalog_manager.cc:4841] TS 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049): tablet 15a605cfd2a04080a59d1e3d88c2d033 (table test-table [id=e96c566a0a0b4220b8512bf38baf69b5]) successfully deleted
I20260924 10:44:32.248594 4487 catalog_manager.cc:4841] TS e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387): tablet 15a605cfd2a04080a59d1e3d88c2d033 (table test-table [id=e96c566a0a0b4220b8512bf38baf69b5]) successfully deleted
I20260924 10:44:32.248875 4779 tablet_service.cc:1467] Processing CreateTablet for tablet d6114e9d213e49fdacdb20352449c504 (DEFAULT_TABLE table=test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:32.249172 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6114e9d213e49fdacdb20352449c504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:32.249555 5056 ts_tablet_manager.cc:1918] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:32.249647 5056 log.cc:1196] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/15a605cfd2a04080a59d1e3d88c2d033
I20260924 10:44:32.249962 5056 ts_tablet_manager.cc:1939] T 15a605cfd2a04080a59d1e3d88c2d033 P 83e3116a3afa4ffca6264d82a1022d43: Deleting consensus metadata
I20260924 10:44:32.250922 4486 catalog_manager.cc:4841] TS 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393): tablet 15a605cfd2a04080a59d1e3d88c2d033 (table test-table [id=e96c566a0a0b4220b8512bf38baf69b5]) successfully deleted
I20260924 10:44:32.250973 5009 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Bootstrap starting.
I20260924 10:44:32.251586 5009 tablet_bootstrap.cc:654] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:32.252653 5012 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: No bootstrap required, opened a new log
I20260924 10:44:32.252756 5012 ts_tablet_manager.cc:1397] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 10:44:32.253322 5009 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: No bootstrap required, opened a new log
I20260924 10:44:32.253399 5009 ts_tablet_manager.cc:1397] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:32.253309 5012 raft_consensus.cc:348] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.253439 5012 raft_consensus.cc:374] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.253473 5012 raft_consensus.cc:729] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.253607 5012 consensus_queue.cc:260] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.253865 5012 ts_tablet_manager.cc:1428] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:32.253845 5009 raft_consensus.cc:348] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.253964 5009 raft_consensus.cc:374] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.254002 5009 raft_consensus.cc:729] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.254136 5009 consensus_queue.cc:260] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.254374 5009 ts_tablet_manager.cc:1428] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:32.255064 5008 tablet_bootstrap.cc:492] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: No bootstrap required, opened a new log
I20260924 10:44:32.255147 5008 ts_tablet_manager.cc:1397] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260924 10:44:32.255580 5008 raft_consensus.cc:348] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.255699 5008 raft_consensus.cc:374] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:32.255733 5008 raft_consensus.cc:729] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Initialized, Role: FOLLOWER
I20260924 10:44:32.255844 5008 consensus_queue.cc:260] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.256108 5008 ts_tablet_manager.cc:1428] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 10:44:32.444505 4986 tablet.cc:2369] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:44:32.486227 4846 tablet.cc:2369] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:32.643213 5030 raft_consensus.cc:482] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:32.643352 5030 raft_consensus.cc:504] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.643755 5030 leader_election.cc:288] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:32.644168 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6114e9d213e49fdacdb20352449c504" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78" is_pre_election: true
I20260924 10:44:32.644184 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6114e9d213e49fdacdb20352449c504" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295" is_pre_election: true
I20260924 10:44:32.644338 4799 raft_consensus.cc:2454] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:32.644362 4659 raft_consensus.cc:2454] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:32.644622 4871 leader_election.cc:302] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e3116a3afa4ffca6264d82a1022d43, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:32.644801 5030 raft_consensus.cc:2781] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:32.644866 5030 raft_consensus.cc:482] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:32.644941 5030 raft_consensus.cc:3037] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.646112 5030 raft_consensus.cc:504] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.646418 5030 leader_election.cc:288] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Requested vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:32.646718 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6114e9d213e49fdacdb20352449c504" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295"
I20260924 10:44:32.646837 4799 raft_consensus.cc:3037] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.647693 4799 raft_consensus.cc:2454] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:32.647917 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6114e9d213e49fdacdb20352449c504" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78"
I20260924 10:44:32.647960 4871 leader_election.cc:302] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e3116a3afa4ffca6264d82a1022d43, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:32.648032 4659 raft_consensus.cc:3037] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:32.648142 5030 raft_consensus.cc:2781] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:32.648231 5030 raft_consensus.cc:686] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 LEADER]: Becoming Leader. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Running, Role: LEADER
I20260924 10:44:32.648378 5030 consensus_queue.cc:237] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:32.649261 4659 raft_consensus.cc:2454] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:32.649878 4501 catalog_manager.cc:5495] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } health_report { overall_health: HEALTHY } } }
I20260924 10:44:32.715718 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.712415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.2.182.3:37393)
0924 10:44:32.712637 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.712642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.712655 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:32.712792 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.712795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.712841 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.712972 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.713254 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.713779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.714634 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.714888 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.715009 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.715263 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.715349 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.715428 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.715523 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.715572 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":47,"threads_started":1}
I20260924 10:44:32.723542 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.720775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:37597 (local address 127.2.182.3:37393)
0924 10:44:32.720882 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:32.720885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:32.721024 (+ 139us) server_negotiation.cc:407] Connection header received
0924 10:44:32.721109 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:32.721111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:32.721165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:32.721223 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:32.721431 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.721879 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.722633 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.722844 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.722918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.723176 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.723228 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.723278 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:32.723354 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:44:32.723428 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":42}
I20260924 10:44:32.723565 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:32.720696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.2.182.3:37597)
0924 10:44:32.720947 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:32.720966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:32.721040 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:32.721244 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:32.721246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:32.721252 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:32.721350 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.721354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.721958 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.721960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.722548 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:32.722555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:32.722957 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:32.722960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:32.723032 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:32.723055 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:32.723296 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:32.723301 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:44:32.723349 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:44:32.723451 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":31,"thread_start_us":72,"threads_started":1}
I20260924 10:44:32.728693 4659 raft_consensus.cc:1261] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.728744 4799 raft_consensus.cc:1261] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:32.729113 5030 consensus_queue.cc:1035] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:32.729246 5014 consensus_queue.cc:1035] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:44:32.742525 5035 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:44:34.293265 4500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:34.293445 4500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:34.295639 4500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Sending DeleteTablet for 3 replicas of tablet d6114e9d213e49fdacdb20352449c504
I20260924 10:44:34.296226 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet d6114e9d213e49fdacdb20352449c504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44042
I20260924 10:44:34.296247 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet d6114e9d213e49fdacdb20352449c504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:39896
I20260924 10:44:34.296228 4919 tablet_service.cc:1514] Processing DeleteTablet for tablet d6114e9d213e49fdacdb20352449c504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:53960
I20260924 10:44:34.296623 5093 tablet_replica.cc:330] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: stopping tablet replica
I20260924 10:44:34.296623 5094 tablet_replica.cc:330] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: stopping tablet replica
I20260924 10:44:34.296762 5093 raft_consensus.cc:2229] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:34.296775 5094 raft_consensus.cc:2229] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:34.296895 5093 raft_consensus.cc:2258] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.297049 5092 tablet_replica.cc:330] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: stopping tablet replica
I20260924 10:44:34.297291 5092 raft_consensus.cc:2229] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:34.297433 5092 raft_consensus.cc:2258] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.297561 5094 raft_consensus.cc:2258] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.298133 5093 ts_tablet_manager.cc:1905] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.298245 5092 ts_tablet_manager.cc:1905] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.300181 5094 ts_tablet_manager.cc:1905] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.301154 5092 ts_tablet_manager.cc:1918] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:44:34.301285 5092 log.cc:1196] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/d6114e9d213e49fdacdb20352449c504
I20260924 10:44:34.301465 5093 ts_tablet_manager.cc:1918] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:44:34.301544 5093 log.cc:1196] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/d6114e9d213e49fdacdb20352449c504
I20260924 10:44:34.301726 5092 ts_tablet_manager.cc:1939] T d6114e9d213e49fdacdb20352449c504 P 3a544f107df24e6f89217ebc18565a78: Deleting consensus metadata
I20260924 10:44:34.301892 5093 ts_tablet_manager.cc:1939] T d6114e9d213e49fdacdb20352449c504 P e25e041baf0e4c5180d70dcdfebc0295: Deleting consensus metadata
I20260924 10:44:34.302440 4488 catalog_manager.cc:4841] TS 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049): tablet d6114e9d213e49fdacdb20352449c504 (table test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]) successfully deleted
I20260924 10:44:34.302551 4487 catalog_manager.cc:4841] TS e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387): tablet d6114e9d213e49fdacdb20352449c504 (table test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]) successfully deleted
I20260924 10:44:34.302954 5094 ts_tablet_manager.cc:1918] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:44:34.303045 5094 log.cc:1196] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/d6114e9d213e49fdacdb20352449c504
I20260924 10:44:34.303290 5094 ts_tablet_manager.cc:1939] T d6114e9d213e49fdacdb20352449c504 P 83e3116a3afa4ffca6264d82a1022d43: Deleting consensus metadata
I20260924 10:44:34.303869 4486 catalog_manager.cc:4841] TS 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393): tablet d6114e9d213e49fdacdb20352449c504 (table test-table [id=dfcc7f4eb05e4f1e9b560cabd75ae8bf]) successfully deleted
Sep 24 10:44:34 dist-test-slave-6lwt krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:44:34 dist-test-slave-6lwt krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:34.327930 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.315381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.2.182.62:36075)
0924 10:44:34.315627 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.315637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.315733 (+ 96us) server_negotiation.cc:407] Connection header received
0924 10:44:34.315907 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.315910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.315963 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.316051 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:34.316682 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.317294 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.318114 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.318349 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.318450 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.324780 (+ 6330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:34.324783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:34.324785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:34.324806 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:34.326370 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:34.326767 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:34.326772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:34.326774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:34.326860 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:34.327122 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:34.327127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:34.327128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:34.327297 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:34.327430 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.327653 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.327767 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260924 10:44:34.330176 4500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:34.330595 4500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:34.334812 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 6a35b584aa0245afb130cacbda418304 (DEFAULT_TABLE table=test-table [id=5278237b17b742969e9c2254e6503e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.334851 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 6a35b584aa0245afb130cacbda418304 (DEFAULT_TABLE table=test-table [id=5278237b17b742969e9c2254e6503e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.335144 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a35b584aa0245afb130cacbda418304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.335182 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a35b584aa0245afb130cacbda418304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.335150 4919 tablet_service.cc:1467] Processing CreateTablet for tablet 6a35b584aa0245afb130cacbda418304 (DEFAULT_TABLE table=test-table [id=5278237b17b742969e9c2254e6503e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.335383 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a35b584aa0245afb130cacbda418304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.337668 5104 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Bootstrap starting.
I20260924 10:44:34.337766 5105 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Bootstrap starting.
I20260924 10:44:34.337777 5103 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Bootstrap starting.
I20260924 10:44:34.338582 5105 tablet_bootstrap.cc:654] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.338678 5103 tablet_bootstrap.cc:654] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.338958 5104 tablet_bootstrap.cc:654] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.339727 5105 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: No bootstrap required, opened a new log
I20260924 10:44:34.339812 5105 ts_tablet_manager.cc:1397] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:34.339985 5103 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: No bootstrap required, opened a new log
I20260924 10:44:34.340072 5103 ts_tablet_manager.cc:1397] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:34.340128 5104 tablet_bootstrap.cc:492] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: No bootstrap required, opened a new log
I20260924 10:44:34.340211 5104 ts_tablet_manager.cc:1397] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:44:34.340245 5105 raft_consensus.cc:348] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.340351 5105 raft_consensus.cc:374] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.340444 5105 raft_consensus.cc:729] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.340525 5103 raft_consensus.cc:348] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.340607 5105 consensus_queue.cc:260] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.340675 5103 raft_consensus.cc:374] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.340627 5104 raft_consensus.cc:348] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.340708 5103 raft_consensus.cc:729] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.340718 5104 raft_consensus.cc:374] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.340754 5104 raft_consensus.cc:729] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.340831 5103 consensus_queue.cc:260] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.340874 5104 consensus_queue.cc:260] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.341216 5104 ts_tablet_manager.cc:1428] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:34.341209 5103 ts_tablet_manager.cc:1428] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:34.341351 5105 ts_tablet_manager.cc:1428] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:34.472337 5084 raft_consensus.cc:482] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:34.472471 5084 raft_consensus.cc:504] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.472987 5084 leader_election.cc:288] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:34.473418 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a35b584aa0245afb130cacbda418304" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78" is_pre_election: true
I20260924 10:44:34.473418 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a35b584aa0245afb130cacbda418304" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295" is_pre_election: true
I20260924 10:44:34.473548 4659 raft_consensus.cc:2454] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:34.473598 4799 raft_consensus.cc:2454] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 0.
I20260924 10:44:34.473924 4872 leader_election.cc:302] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, 83e3116a3afa4ffca6264d82a1022d43; no voters:
I20260924 10:44:34.474107 5084 raft_consensus.cc:2781] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:34.474185 5084 raft_consensus.cc:482] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:34.474261 5084 raft_consensus.cc:3037] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.475250 5084 raft_consensus.cc:504] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.475584 5084 leader_election.cc:288] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Requested vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387)
I20260924 10:44:34.475903 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a35b584aa0245afb130cacbda418304" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25e041baf0e4c5180d70dcdfebc0295"
I20260924 10:44:34.475911 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a35b584aa0245afb130cacbda418304" candidate_uuid: "83e3116a3afa4ffca6264d82a1022d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78"
I20260924 10:44:34.475988 4799 raft_consensus.cc:3037] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.476032 4659 raft_consensus.cc:3037] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.476941 4799 raft_consensus.cc:2454] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:34.477061 4659 raft_consensus.cc:2454] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e3116a3afa4ffca6264d82a1022d43 in term 1.
I20260924 10:44:34.477239 4871 leader_election.cc:302] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e3116a3afa4ffca6264d82a1022d43, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:34.477429 5084 raft_consensus.cc:2781] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:34.477550 5084 raft_consensus.cc:686] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 LEADER]: Becoming Leader. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Running, Role: LEADER
I20260924 10:44:34.477691 5084 consensus_queue.cc:237] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.479314 4501 catalog_manager.cc:5495] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "83e3116a3afa4ffca6264d82a1022d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } health_report { overall_health: HEALTHY } } }
I20260924 10:44:34.498558 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.495594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.2.182.3:37393)
0924 10:44:34.495833 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.495840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.495852 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:34.495946 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.495951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.496000 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.496079 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:34.496280 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.496778 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.497642 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.497874 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.497951 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.498149 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.498212 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.498261 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.498339 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.498381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260924 10:44:34.503707 4799 raft_consensus.cc:1261] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:34.503729 4659 raft_consensus.cc:1261] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Refusing update from remote peer 83e3116a3afa4ffca6264d82a1022d43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:34.504065 5075 consensus_queue.cc:1035] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:34.504201 5084 consensus_queue.cc:1035] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:34.537426 4501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:34.537586 4501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:34.539570 4501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Sending DeleteTablet for 3 replicas of tablet 6a35b584aa0245afb130cacbda418304
I20260924 10:44:34.540149 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a35b584aa0245afb130cacbda418304 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44042
I20260924 10:44:34.540261 4918 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a35b584aa0245afb130cacbda418304 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:53960
I20260924 10:44:34.540343 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a35b584aa0245afb130cacbda418304 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:34 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:39896
I20260924 10:44:34.540457 5094 tablet_replica.cc:330] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: stopping tablet replica
I20260924 10:44:34.540495 5092 tablet_replica.cc:330] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: stopping tablet replica
I20260924 10:44:34.540499 5093 tablet_replica.cc:330] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: stopping tablet replica
I20260924 10:44:34.540570 5094 raft_consensus.cc:2229] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:34.540591 5092 raft_consensus.cc:2229] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:34.540621 5093 raft_consensus.cc:2229] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:34.540692 5092 raft_consensus.cc:2258] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.540735 5093 raft_consensus.cc:2258] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.540750 5094 raft_consensus.cc:2258] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:34.540872 4501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:34.541268 4501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:34.541445 5094 ts_tablet_manager.cc:1905] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.541546 5093 ts_tablet_manager.cc:1905] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.543890 5094 ts_tablet_manager.cc:1918] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:34.543890 5093 ts_tablet_manager.cc:1918] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:34.543969 5094 log.cc:1196] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/6a35b584aa0245afb130cacbda418304
I20260924 10:44:34.543969 5093 log.cc:1196] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/6a35b584aa0245afb130cacbda418304
I20260924 10:44:34.544248 5093 ts_tablet_manager.cc:1939] T 6a35b584aa0245afb130cacbda418304 P e25e041baf0e4c5180d70dcdfebc0295: Deleting consensus metadata
I20260924 10:44:34.544271 5094 ts_tablet_manager.cc:1939] T 6a35b584aa0245afb130cacbda418304 P 83e3116a3afa4ffca6264d82a1022d43: Deleting consensus metadata
I20260924 10:44:34.544833 4486 catalog_manager.cc:4841] TS 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393): tablet 6a35b584aa0245afb130cacbda418304 (table test-table [id=5278237b17b742969e9c2254e6503e2f]) successfully deleted
I20260924 10:44:34.545111 5092 ts_tablet_manager.cc:1905] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:34.547621 5092 ts_tablet_manager.cc:1918] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:44:34.547710 5092 log.cc:1196] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/6a35b584aa0245afb130cacbda418304
I20260924 10:44:34.548434 4779 tablet_service.cc:1467] Processing CreateTablet for tablet c04497011a784509a798f9e7c767cb1e (DEFAULT_TABLE table=test-table [id=4b9efa99b12647e59036b4b027c49fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.548597 4918 tablet_service.cc:1467] Processing CreateTablet for tablet c04497011a784509a798f9e7c767cb1e (DEFAULT_TABLE table=test-table [id=4b9efa99b12647e59036b4b027c49fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.548678 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04497011a784509a798f9e7c767cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.548817 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04497011a784509a798f9e7c767cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.550841 5104 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Bootstrap starting.
I20260924 10:44:34.550977 4639 tablet_service.cc:1467] Processing CreateTablet for tablet c04497011a784509a798f9e7c767cb1e (DEFAULT_TABLE table=test-table [id=4b9efa99b12647e59036b4b027c49fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:34.551095 5105 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Bootstrap starting.
I20260924 10:44:34.551198 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04497011a784509a798f9e7c767cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:34.551674 5104 tablet_bootstrap.cc:654] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.551831 5105 tablet_bootstrap.cc:654] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.552932 5104 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: No bootstrap required, opened a new log
I20260924 10:44:34.553025 5104 ts_tablet_manager.cc:1397] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:44:34.553335 5103 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78: Bootstrap starting.
I20260924 10:44:34.547962 5092 ts_tablet_manager.cc:1939] T 6a35b584aa0245afb130cacbda418304 P 3a544f107df24e6f89217ebc18565a78: Deleting consensus metadata
I20260924 10:44:34.553490 5104 raft_consensus.cc:348] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.553598 5104 raft_consensus.cc:374] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.553637 5104 raft_consensus.cc:729] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.553771 5104 consensus_queue.cc:260] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.554019 5104 ts_tablet_manager.cc:1428] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:44:34.554157 5103 tablet_bootstrap.cc:654] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:34.554545 4488 catalog_manager.cc:4841] TS 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049): tablet 6a35b584aa0245afb130cacbda418304 (table test-table [id=5278237b17b742969e9c2254e6503e2f]) successfully deleted
I20260924 10:44:34.554730 4487 catalog_manager.cc:4841] TS e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387): tablet 6a35b584aa0245afb130cacbda418304 (table test-table [id=5278237b17b742969e9c2254e6503e2f]) successfully deleted
I20260924 10:44:34.554749 5105 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: No bootstrap required, opened a new log
I20260924 10:44:34.554831 5105 ts_tablet_manager.cc:1397] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:44:34.555209 5105 raft_consensus.cc:348] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.555286 5105 raft_consensus.cc:374] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.555307 5105 raft_consensus.cc:729] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3116a3afa4ffca6264d82a1022d43, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.555371 5103 tablet_bootstrap.cc:492] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78: No bootstrap required, opened a new log
I20260924 10:44:34.555442 5103 ts_tablet_manager.cc:1397] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:34.555426 5105 consensus_queue.cc:260] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.555692 5105 ts_tablet_manager.cc:1428] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:34.555790 5103 raft_consensus.cc:348] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.555871 5103 raft_consensus.cc:374] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:34.555894 5103 raft_consensus.cc:729] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a544f107df24e6f89217ebc18565a78, State: Initialized, Role: FOLLOWER
I20260924 10:44:34.555980 5103 consensus_queue.cc:260] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.556203 5103 ts_tablet_manager.cc:1428] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:44:34.702521 5110 raft_consensus.cc:482] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:34.702656 5110 raft_consensus.cc:504] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.703676 5110 leader_election.cc:288] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393)
I20260924 10:44:34.707728 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.704127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:40475 (local address 127.2.182.3:37393)
0924 10:44:34.704254 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.704259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.704454 (+ 195us) server_negotiation.cc:407] Connection header received
0924 10:44:34.704586 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.704591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.704652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.704724 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:34.705063 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.705617 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.706777 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.707177 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.707276 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.707344 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.707524 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.707602 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:44:34.707747 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.703918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.2.182.2:40475)
0924 10:44:34.704360 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:34.704372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:34.704466 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:34.704759 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:34.704763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:34.704784 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:34.704962 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.704969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.705724 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.705730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.706655 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.706663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.707294 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.707298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.707391 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.707419 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:44:34.707489 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:44:34.707576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":59,"thread_start_us":70,"threads_started":1}
I20260924 10:44:34.707788 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.704014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:57141 (local address 127.2.182.1:46049)
0924 10:44:34.704254 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.704268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.704543 (+ 275us) server_negotiation.cc:407] Connection header received
0924 10:44:34.704672 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.704674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.704747 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.704831 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:34.705167 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.705707 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.706778 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.707192 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.707324 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.707386 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.707583 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.707661 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260924 10:44:34.707762 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.703917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:46049 (local address 127.2.182.2:57141)
0924 10:44:34.704405 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:34.704438 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:34.704572 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:34.704857 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:34.704860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:34.704881 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:34.705053 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.705060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.705833 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.705836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.706649 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.706656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.707326 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.707329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.707441 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.707458 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:44:34.707550 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:44:34.707611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":35,"thread_start_us":105,"threads_started":1}
I20260924 10:44:34.708231 4658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04497011a784509a798f9e7c767cb1e" candidate_uuid: "e25e041baf0e4c5180d70dcdfebc0295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78" is_pre_election: true
I20260924 10:44:34.708405 4658 raft_consensus.cc:2454] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e25e041baf0e4c5180d70dcdfebc0295 in term 0.
I20260924 10:44:34.708428 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04497011a784509a798f9e7c767cb1e" candidate_uuid: "e25e041baf0e4c5180d70dcdfebc0295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3116a3afa4ffca6264d82a1022d43" is_pre_election: true
I20260924 10:44:34.708581 4939 raft_consensus.cc:2454] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e25e041baf0e4c5180d70dcdfebc0295 in term 0.
I20260924 10:44:34.708797 4732 leader_election.cc:302] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:34.709089 5110 raft_consensus.cc:2781] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:34.709149 5110 raft_consensus.cc:482] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:34.709177 5110 raft_consensus.cc:3037] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.710196 5110 raft_consensus.cc:504] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.710553 5110 leader_election.cc:288] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [CANDIDATE]: Term 1 election: Requested vote from peers 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049), 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393)
I20260924 10:44:34.710835 4658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04497011a784509a798f9e7c767cb1e" candidate_uuid: "e25e041baf0e4c5180d70dcdfebc0295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a544f107df24e6f89217ebc18565a78"
I20260924 10:44:34.710891 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04497011a784509a798f9e7c767cb1e" candidate_uuid: "e25e041baf0e4c5180d70dcdfebc0295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3116a3afa4ffca6264d82a1022d43"
I20260924 10:44:34.710992 4658 raft_consensus.cc:3037] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.711030 4939 raft_consensus.cc:3037] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:34.711895 4658 raft_consensus.cc:2454] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e25e041baf0e4c5180d70dcdfebc0295 in term 1.
I20260924 10:44:34.711880 4939 raft_consensus.cc:2454] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e25e041baf0e4c5180d70dcdfebc0295 in term 1.
I20260924 10:44:34.712162 4732 leader_election.cc:302] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a544f107df24e6f89217ebc18565a78, e25e041baf0e4c5180d70dcdfebc0295; no voters:
I20260924 10:44:34.712342 5110 raft_consensus.cc:2781] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:34.712561 5110 raft_consensus.cc:686] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 1 LEADER]: Becoming Leader. State: Replica: e25e041baf0e4c5180d70dcdfebc0295, State: Running, Role: LEADER
I20260924 10:44:34.712807 5110 consensus_queue.cc:237] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } }
I20260924 10:44:34.715132 4501 catalog_manager.cc:5495] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 reported cstate change: term changed from 0 to 1, leader changed from <none> to e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "e25e041baf0e4c5180d70dcdfebc0295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e25e041baf0e4c5180d70dcdfebc0295" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 } health_report { overall_health: UNKNOWN } } }
I20260924 10:44:34.749269 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.745189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.2.182.2:42387)
0924 10:44:34.745517 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.745527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.745541 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:44:34.745620 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.745622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.745675 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.745801 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:34.746037 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.746650 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.747481 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.747733 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.747844 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.748115 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.748562 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.748706 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.748741 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.748790 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":140,"threads_started":1}
I20260924 10:44:34.759238 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.756000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37393 (local address 127.2.182.2:39845)
0924 10:44:34.756321 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:34.756332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:34.756444 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:34.756694 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:34.756697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:34.756706 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:34.756819 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.756824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.757526 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.757531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.758151 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.758158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.758580 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.758584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.758678 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.758713 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.758978 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.758986 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:44:34.759043 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 10:44:34.759110 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260924 10:44:34.759229 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.756127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:39845 (local address 127.2.182.3:37393)
0924 10:44:34.756269 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.756273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.756405 (+ 132us) server_negotiation.cc:407] Connection header received
0924 10:44:34.756550 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.756553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.756606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.756683 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:34.756949 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.757420 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.758260 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.758465 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.758570 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.758833 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.758900 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.758972 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.759050 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.759099 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:44:34.764549 4939 raft_consensus.cc:1261] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Refusing update from remote peer e25e041baf0e4c5180d70dcdfebc0295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:34.764616 4658 raft_consensus.cc:1261] T c04497011a784509a798f9e7c767cb1e P 3a544f107df24e6f89217ebc18565a78 [term 1 FOLLOWER]: Refusing update from remote peer e25e041baf0e4c5180d70dcdfebc0295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:34.765077 5124 consensus_queue.cc:1035] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a544f107df24e6f89217ebc18565a78" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:34.765282 5110 consensus_queue.cc:1035] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e3116a3afa4ffca6264d82a1022d43" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:34.780251 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.777113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:51923 (local address 127.2.182.2:42387)
0924 10:44:34.777215 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:34.777222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:34.777387 (+ 165us) server_negotiation.cc:407] Connection header received
0924 10:44:34.777521 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:34.777524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:34.777565 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:34.777648 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:34.777908 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.778380 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.779259 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.779485 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.779587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.779818 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.779898 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.779996 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:34.780082 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:44:34.780129 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 10:44:34.780282 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:34.777070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42387 (local address 127.2.182.3:51923)
0924 10:44:34.777299 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:34.777317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:34.777428 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:34.777658 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:34.777661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:34.777669 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:44:34.777772 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.777777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.778472 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.778474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.779143 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:34.779151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:34.779587 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:34.779590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:34.779673 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:34.779706 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:44:34.780000 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:44:34.780010 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:44:34.780074 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:44:34.780150 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
W20260924 10:44:34.781196 5076 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:44:36.334401 4502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:36.334542 4502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:36.336935 4502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fed902b70be94dc780f34e1cf45f9408: Sending DeleteTablet for 3 replicas of tablet c04497011a784509a798f9e7c767cb1e
I20260924 10:44:36.337509 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet c04497011a784509a798f9e7c767cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:36 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44042
I20260924 10:44:36.337517 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet c04497011a784509a798f9e7c767cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:36 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:39896
I20260924 10:44:36.337594 4918 tablet_service.cc:1514] Processing DeleteTablet for tablet c04497011a784509a798f9e7c767cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:36 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:53960
I20260924 10:44:36.337819 5158 tablet_replica.cc:330] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: stopping tablet replica
I20260924 10:44:36.337877 5160 tablet_replica.cc:330] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: stopping tablet replica
I20260924 10:44:36.337920 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4574
I20260924 10:44:36.337946 5158 raft_consensus.cc:2229] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:36.338617 5160 raft_consensus.cc:2229] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:36.338757 5160 raft_consensus.cc:2258] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:36.339284 5158 raft_consensus.cc:2258] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:36.339427 5160 ts_tablet_manager.cc:1905] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:36.339954 5158 ts_tablet_manager.cc:1905] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:36.341894 5160 ts_tablet_manager.cc:1918] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:44:36.341988 5160 log.cc:1196] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/c04497011a784509a798f9e7c767cb1e
I20260924 10:44:36.342242 5160 ts_tablet_manager.cc:1939] T c04497011a784509a798f9e7c767cb1e P 83e3116a3afa4ffca6264d82a1022d43: Deleting consensus metadata
I20260924 10:44:36.343029 4486 catalog_manager.cc:4841] TS 83e3116a3afa4ffca6264d82a1022d43 (127.2.182.3:37393): tablet c04497011a784509a798f9e7c767cb1e (table test-table [id=4b9efa99b12647e59036b4b027c49fa5]) successfully deleted
I20260924 10:44:36.343184 5158 ts_tablet_manager.cc:1918] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:44:36.343266 5158 log.cc:1196] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/c04497011a784509a798f9e7c767cb1e
I20260924 10:44:36.343567 5158 ts_tablet_manager.cc:1939] T c04497011a784509a798f9e7c767cb1e P e25e041baf0e4c5180d70dcdfebc0295: Deleting consensus metadata
I20260924 10:44:36.344237 4487 catalog_manager.cc:4841] TS e25e041baf0e4c5180d70dcdfebc0295 (127.2.182.2:42387): tablet c04497011a784509a798f9e7c767cb1e (table test-table [id=4b9efa99b12647e59036b4b027c49fa5]) successfully deleted
W20260924 10:44:36.347450 4488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:46049 (error 108)
I20260924 10:44:36.347725 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4714
I20260924 10:44:36.350164 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:36.348132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:46049 (local address 127.0.0.1:45128)
0924 10:44:36.349962 (+ 1830us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:36.350044 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1734,"thread_start_us":75,"threads_started":1}
W20260924 10:44:36.350311 4488 catalog_manager.cc:4584] TS 3a544f107df24e6f89217ebc18565a78 (127.2.182.1:46049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c04497011a784509a798f9e7c767cb1e: Network error: Client connection negotiation failed: client connection to 127.2.182.1:46049: connect: Connection refused (error 111)
I20260924 10:44:36.355096 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4854
I20260924 10:44:36.361666 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 4470
2026-09-24T10:44:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6382 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:36 dist-test-slave-6lwt krb5kdc[5165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:36 dist-test-slave-6lwt krb5kdc[5165](info): set up 2 sockets
Sep 24 10:44:36 dist-test-slave-6lwt krb5kdc[5165](info): commencing operation
krb5kdc: starting...
W20260924 10:44:37.381100 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:37 dist-test-slave-6lwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:37.497740 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36243
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:37.604312 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:37.604550 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:37.604589 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:37.608517 5178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:37.608566 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:37.608604 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:37.608620 5178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:37.608641 5178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:37.613668 5178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:37.614548 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:37.615458 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:37.624141 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:37.624219 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:37.624190 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:37.623971 5178 server_base.cc:1034] running on GCE node
I20260924 10:44:37.624992 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:37.626092 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:37.627245 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790246677627234 us; error 28 us; skew 500 ppm
Sep 24 10:44:37 dist-test-slave-6lwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:37.629913 5178 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:37.630700 5178 webserver.cc:460] Webserver started at http://127.2.182.62:36109/ using document root <none> and password file <none>
I20260924 10:44:37.631228 5178 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:37.631319 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:37.631551 5178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:37.633544 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "25de53a814154952a81b35a68ed451d2"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "a3e05ad583530573cb6ae542602f2a07"
server_key_iv: "e827d9141eea46ab2399e279558c5965"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.634087 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "25de53a814154952a81b35a68ed451d2"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "a3e05ad583530573cb6ae542602f2a07"
server_key_iv: "e827d9141eea46ab2399e279558c5965"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.637846 5178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:37.640055 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:37.641280 5178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:37.641418 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "25de53a814154952a81b35a68ed451d2"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "a3e05ad583530573cb6ae542602f2a07"
server_key_iv: "e827d9141eea46ab2399e279558c5965"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.641521 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:37.654259 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:37.657393 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:37.657577 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:37.664064 5178 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36243
I20260924 10:44:37.664132 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36243 every 8 connection(s)
I20260924 10:44:37.665030 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:37.667593 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:37.672601 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2: Bootstrap starting.
I20260924 10:44:37.672995 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5178
I20260924 10:44:37.673107 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:37.673370 2776 external_mini_cluster.cc:1428] Setting key 89ca70ffa9792f59e140cf684a05002d
I20260924 10:44:37.675076 5246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:37.675895 5246 log.cc:824] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:37.677857 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2: No bootstrap required, opened a new log
I20260924 10:44:37.681704 5246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } }
I20260924 10:44:37.681988 5246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:37.682034 5246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25de53a814154952a81b35a68ed451d2, State: Initialized, Role: FOLLOWER
Sep 24 10:44:37 dist-test-slave-6lwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:37.682499 5246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } }
I20260924 10:44:37.682655 5246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:37.682729 5246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:37.682838 5246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:37.683804 5246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } }
I20260924 10:44:37.684165 5246 leader_election.cc:302] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25de53a814154952a81b35a68ed451d2; no voters:
I20260924 10:44:37.684474 5246 leader_election.cc:288] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:37.684537 5251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:37.684737 5251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [term 1 LEADER]: Becoming Leader. State: Replica: 25de53a814154952a81b35a68ed451d2, State: Running, Role: LEADER
I20260924 10:44:37.685084 5251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } }
I20260924 10:44:37.685647 5246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:37.686167 5251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25de53a814154952a81b35a68ed451d2. Latest consensus state: current_term: 1 leader_uuid: "25de53a814154952a81b35a68ed451d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } } }
I20260924 10:44:37.686285 5251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:37.686568 5252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25de53a814154952a81b35a68ed451d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25de53a814154952a81b35a68ed451d2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36243 } } }
I20260924 10:44:37.686652 5252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:37.686684 5255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:37.688580 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:37.674643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.2.182.62:36243)
0924 10:44:37.675070 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:37.675090 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:37.675134 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:44:37.675928 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:37.675949 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:37.676336 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:37.676642 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:37.677320 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.678739 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.679549 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.680250 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.680455 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:37.683060 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:37.683065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:37.683076 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:37.683099 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:37.685002 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:37.685481 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:37.685485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:37.685489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:37.685586 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:37.685904 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:37.685908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:37.685909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:37.686916 (+ 1007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:37.687091 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:37.687461 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:44:37.687735 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260924 10:44:37.689643 5255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:37.694272 5255 catalog_manager.cc:1347] Generated new cluster ID: d8f2ab86636b4bc6b3b2288492fd1b56
I20260924 10:44:37.694339 5255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:37.708233 5255 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:37.709381 5255 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:37.716547 5255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25de53a814154952a81b35a68ed451d2: Generated new TSK 0
I20260924 10:44:37.717283 5255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:37.749778 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:37.856745 5272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:37.856983 5272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:37.857023 5272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:37.860729 5272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:37.860775 5272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:37.860827 5272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:37.866796 5272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:37.867635 5272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:37.868508 5272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:37.877537 5278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:37.877578 5272 server_base.cc:1034] running on GCE node
W20260924 10:44:37.877542 5277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:37.877544 5280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:37.878180 5272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:37.878712 5272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:37.879868 5272 hybrid_clock.cc:648] HybridClock initialized: now 1790246677879836 us; error 46 us; skew 500 ppm
Sep 24 10:44:37 dist-test-slave-6lwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:37.882539 5272 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:37.883260 5272 webserver.cc:460] Webserver started at http://127.2.182.1:36197/ using document root <none> and password file <none>
I20260924 10:44:37.883785 5272 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:37.883842 5272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:37.883988 5272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:37.885972 5272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "60ef4068599e4b718af3b2de8c99e085"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "16a4475b7a38c85fa6328e3d13a9bcc6"
server_key_iv: "d53f7d03d36ac12071206be9c50a9de0"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.886488 5272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "60ef4068599e4b718af3b2de8c99e085"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "16a4475b7a38c85fa6328e3d13a9bcc6"
server_key_iv: "d53f7d03d36ac12071206be9c50a9de0"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.889954 5272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:37.892060 5287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:37.893282 5272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:37.893371 5272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "60ef4068599e4b718af3b2de8c99e085"
format_stamp: "Formatted at 2026-09-24 10:44:37 on dist-test-slave-6lwt"
server_key: "16a4475b7a38c85fa6328e3d13a9bcc6"
server_key_iv: "d53f7d03d36ac12071206be9c50a9de0"
server_key_version: "encryptionkey@0"
I20260924 10:44:37.893443 5272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:37.908427 5272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:37.911116 5272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:37.911250 5272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:37.911684 5272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:37.912358 5272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:37.912401 5272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:37.912436 5272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:37.912451 5272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:37.921075 5272 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:43159
I20260924 10:44:37.921151 5400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:43159 every 8 connection(s)
I20260924 10:44:37.921922 5272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:37.925321 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5272
I20260924 10:44:37.925408 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:37.925649 2776 external_mini_cluster.cc:1428] Setting key 3c8e6d715012e2758c18a417398396ec
Sep 24 10:44:37 dist-test-slave-6lwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:37.935770 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:37.923554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:38279 (local address 127.2.182.62:36243)
0924 10:44:37.923711 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:37.923716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:37.924470 (+ 754us) server_negotiation.cc:407] Connection header received
0924 10:44:37.925821 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:37.925824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:37.925880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:37.925984 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:37.927214 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.927764 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.929232 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.929463 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.929546 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:37.931990 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:37.931997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:37.932002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:37.932036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:37.933766 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:37.934243 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:37.934249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:37.934254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:37.934343 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:37.934655 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:37.934659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:37.934661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:37.934924 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:37.935034 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:37.935500 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 10:44:37.935633 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:44:37.936612 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:37.923903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36243 (local address 127.2.182.1:38279)
0924 10:44:37.924329 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:37.924371 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:37.925596 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:37.926140 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:37.926146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:37.926570 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:37.927073 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:37.927085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.927894 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.927897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:37.929108 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:37.929121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:37.929612 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:37.929617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:37.929846 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:37.930419 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:37.930435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:37.931824 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:37.933881 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:37.933885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:37.933896 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:37.934141 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:37.934434 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:37.934436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:37.934438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:37.934558 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:37.935032 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:37.935037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:37.935233 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 10:44:37.935433 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 10:44:37.935680 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:44:37.937911 5401 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36243
I20260924 10:44:37.938122 5401 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:37.938782 5401 heartbeater.cc:511] Master 127.2.182.62:36243 requested a full tablet report, sending...
I20260924 10:44:37.940337 5210 ts_manager.cc:194] Registered new tserver with Master: 60ef4068599e4b718af3b2de8c99e085 (127.2.182.1:43159)
I20260924 10:44:37.941537 5210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:38279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:37.952374 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:38.057027 5406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:38.057269 5406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:38.057343 5406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:38.061198 5406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:38.061264 5406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:38.061375 5406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:38.066339 5406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:38.067283 5406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:38.068228 5406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:38.077376 5414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:38.077376 5411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:38.077448 5406 server_base.cc:1034] running on GCE node
W20260924 10:44:38.077380 5412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:38.078016 5406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:38.078598 5406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:38.079775 5406 hybrid_clock.cc:648] HybridClock initialized: now 1790246678079747 us; error 37 us; skew 500 ppm
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246678, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:38.085951 5406 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:38.086781 5406 webserver.cc:460] Webserver started at http://127.2.182.2:35733/ using document root <none> and password file <none>
I20260924 10:44:38.087359 5406 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:38.087455 5406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:38.087651 5406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:38.089663 5406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "024d8adb85264bce9aa74daaf19f4696"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "80ce82dc105aed9b808bbd5cc5e93468"
server_key_iv: "f846afb70eff23400280bad5a7755009"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.090138 5406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "024d8adb85264bce9aa74daaf19f4696"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "80ce82dc105aed9b808bbd5cc5e93468"
server_key_iv: "f846afb70eff23400280bad5a7755009"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.093592 5406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:44:38.095818 5421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.097222 5406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 10:44:38.097352 5406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "024d8adb85264bce9aa74daaf19f4696"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "80ce82dc105aed9b808bbd5cc5e93468"
server_key_iv: "f846afb70eff23400280bad5a7755009"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.097455 5406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:38.118237 5406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:38.121235 5406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:38.121408 5406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:38.121925 5406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:38.122627 5406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:38.122675 5406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.122736 5406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:38.122772 5406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.131225 5406 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:40107
I20260924 10:44:38.131283 5534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:40107 every 8 connection(s)
I20260924 10:44:38.132077 5406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:38.137902 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5406
I20260924 10:44:38.138002 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:38.138253 2776 external_mini_cluster.cc:1428] Setting key aae4a8f63a70c7b1aaa19776efc31e42
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246678, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:38.145308 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:38.133704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:56039 (local address 127.2.182.62:36243)
0924 10:44:38.133825 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:38.133828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:38.134964 (+ 1136us) server_negotiation.cc:407] Connection header received
0924 10:44:38.135744 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:38.135746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:38.135793 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:38.135867 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:38.136853 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.137429 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.138600 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.138864 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.138959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:38.141379 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:38.141385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:38.141389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:38.141415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:38.142976 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:38.143459 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:38.143465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:38.143470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:38.143563 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:38.143980 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:38.143988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:38.143991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:38.144281 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:38.144416 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:38.144915 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 10:44:38.145064 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:44:38.146035 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:38.133986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36243 (local address 127.2.182.2:56039)
0924 10:44:38.134820 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:38.134859 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:38.135567 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:38.136022 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:38.136028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:38.136315 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:38.136729 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:38.136740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.137536 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.137539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:38.138484 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:38.138492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.138986 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.138990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:38.139146 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:38.139723 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:38.139741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:38.141232 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:38.143087 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:38.143091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:38.143101 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:38.143358 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:38.143679 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:38.143682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:38.143684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:38.143829 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:38.144404 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:38.144408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:38.144652 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 10:44:38.144834 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:44:38.145127 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":536,"threads_started":1}
I20260924 10:44:38.147115 5535 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36243
I20260924 10:44:38.147362 5535 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:38.147931 5535 heartbeater.cc:511] Master 127.2.182.62:36243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:44:38.149076 5210 ts_manager.cc:194] Registered new tserver with Master: 024d8adb85264bce9aa74daaf19f4696 (127.2.182.2:40107)
I20260924 10:44:38.149611 5210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:56039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:38.168603 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:38.282073 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:38.282301 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:38.282342 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:38.286259 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:38.286304 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:38.286357 5540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:38.291218 5540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:38.292032 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:38.292868 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:38.301610 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:38.301640 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:38.301638 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:38.301918 5540 server_base.cc:1034] running on GCE node
I20260924 10:44:38.302317 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:38.302896 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:38.304093 5540 hybrid_clock.cc:648] HybridClock initialized: now 1790246678304086 us; error 27 us; skew 500 ppm
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246678, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:38.306659 5540 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:38.307354 5540 webserver.cc:460] Webserver started at http://127.2.182.3:33307/ using document root <none> and password file <none>
I20260924 10:44:38.307806 5540 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:38.307866 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:38.308012 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:38.309819 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "047fe6b245234fbf8e5df7e9d0b9b371"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "24b35f344a2263994ac1ee16fcfb6aae"
server_key_iv: "999bc5c6afd9f8ff5dfa33894293df27"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.310293 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "047fe6b245234fbf8e5df7e9d0b9b371"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "24b35f344a2263994ac1ee16fcfb6aae"
server_key_iv: "999bc5c6afd9f8ff5dfa33894293df27"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.313582 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:38.315747 5555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.316798 5540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:38.316953 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "047fe6b245234fbf8e5df7e9d0b9b371"
format_stamp: "Formatted at 2026-09-24 10:44:38 on dist-test-slave-6lwt"
server_key: "24b35f344a2263994ac1ee16fcfb6aae"
server_key_iv: "999bc5c6afd9f8ff5dfa33894293df27"
server_key_version: "encryptionkey@0"
I20260924 10:44:38.317049 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:38.347750 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:38.350381 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:38.350576 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:38.351114 5540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:38.351862 5540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:38.351905 5540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.351969 5540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:38.352008 5540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:38.360344 5540 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:39437
I20260924 10:44:38.360373 5668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:39437 every 8 connection(s)
I20260924 10:44:38.361272 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:38.364573 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5540
I20260924 10:44:38.364689 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:38.365006 2776 external_mini_cluster.cc:1428] Setting key 0e99751e600849b360ebc43cd6d14084
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246678, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:38.373783 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:38.362856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:38365 (local address 127.2.182.62:36243)
0924 10:44:38.363015 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:38.363018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:38.363757 (+ 739us) server_negotiation.cc:407] Connection header received
0924 10:44:38.365217 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:38.365221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:38.365292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:38.365390 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:38.366606 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.367184 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.368238 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.368447 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.368517 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:38.370728 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:38.370730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:38.370732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:38.370752 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:38.372131 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:38.372537 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:38.372539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:38.372540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:38.372610 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:38.372878 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:38.372880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:38.372881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:38.373075 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:38.373148 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:38.373578 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 10:44:38.373670 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:44:38.374650 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:38.363215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36243 (local address 127.2.182.3:38365)
0924 10:44:38.363620 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:38.363658 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:38.365014 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:38.365556 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:38.365563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:38.365956 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:38.366471 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:38.366488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.367297 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.367299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:38.368121 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:38.368130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:38.368565 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:38.368568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:38.368688 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:38.369210 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:38.369225 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:38.370595 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:38.372218 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:38.372221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:38.372231 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:38.372459 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:38.372681 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:38.372683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:38.372685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:38.372796 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:38.373159 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:38.373164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:38.373369 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 10:44:38.373532 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 10:44:38.373753 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260924 10:44:38.375838 5669 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36243
I20260924 10:44:38.376060 5669 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:38.376493 5669 heartbeater.cc:511] Master 127.2.182.62:36243 requested a full tablet report, sending...
I20260924 10:44:38.377478 5210 ts_manager.cc:194] Registered new tserver with Master: 047fe6b245234fbf8e5df7e9d0b9b371 (127.2.182.3:39437)
I20260924 10:44:38.378062 5210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:38365
I20260924 10:44:38.378659 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:44:38.390693 5249 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:38.389521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.2.182.62:36243)
0924 10:44:38.389664 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:38.389668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:38.389794 (+ 126us) server_negotiation.cc:407] Connection header received
0924 10:44:38.389875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:38.389877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:38.389911 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:38.389972 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:38.390582 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50100: BlockingRecv error: recv got EOF from 127.0.0.1:50100 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:38.391520 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5272
I20260924 10:44:38.397075 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5406
I20260924 10:44:38.401726 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5540
I20260924 10:44:38.406700 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5178
2026-09-24T10:44:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2043 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5681](info): set up 2 sockets
Sep 24 10:44:38 dist-test-slave-6lwt krb5kdc[5681](info): commencing operation
krb5kdc: starting...
W20260924 10:44:39.429121 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:39.541867 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42569
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:42569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:39.645509 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:39.645769 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:39.645864 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:39.649741 5694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:39.649820 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:39.649863 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:39.649921 5694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:39.649961 5694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:39.654945 5694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:42569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:39.655887 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:39.656795 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:39.664824 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:39.664801 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:39.664856 5694 server_base.cc:1034] running on GCE node
W20260924 10:44:39.664811 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:39.665596 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:39.666580 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:39.667740 5694 hybrid_clock.cc:648] HybridClock initialized: now 1790246679667704 us; error 48 us; skew 500 ppm
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:39.676069 5694 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:39.677016 5694 webserver.cc:460] Webserver started at http://127.2.182.62:34451/ using document root <none> and password file <none>
I20260924 10:44:39.677567 5694 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:39.677623 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:39.677840 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:39.679719 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "d30d699d5e024fb0a25417b049f81ece"
format_stamp: "Formatted at 2026-09-24 10:44:39 on dist-test-slave-6lwt"
server_key: "aa3888b700ffa5839613becb6dc1ba9e"
server_key_iv: "6fc209c6e1d30063a903a2f60d093a0e"
server_key_version: "encryptionkey@0"
I20260924 10:44:39.680255 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "d30d699d5e024fb0a25417b049f81ece"
format_stamp: "Formatted at 2026-09-24 10:44:39 on dist-test-slave-6lwt"
server_key: "aa3888b700ffa5839613becb6dc1ba9e"
server_key_iv: "6fc209c6e1d30063a903a2f60d093a0e"
server_key_version: "encryptionkey@0"
I20260924 10:44:39.683790 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:39.686270 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:39.687429 5694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:39.687577 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "d30d699d5e024fb0a25417b049f81ece"
format_stamp: "Formatted at 2026-09-24 10:44:39 on dist-test-slave-6lwt"
server_key: "aa3888b700ffa5839613becb6dc1ba9e"
server_key_iv: "6fc209c6e1d30063a903a2f60d093a0e"
server_key_version: "encryptionkey@0"
I20260924 10:44:39.687687 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:39.718595 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:39.721457 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:39.721668 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:39.728842 5694 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:42569
I20260924 10:44:39.729125 5761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:42569 every 8 connection(s)
I20260924 10:44:39.729831 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:39.732405 5762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:39.737303 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5694
I20260924 10:44:39.737434 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:39.737617 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece: Bootstrap starting.
I20260924 10:44:39.737738 2776 external_mini_cluster.cc:1428] Setting key 8012a29d2ad58fa9bc3994e147eb90b4
I20260924 10:44:39.740132 5762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:39.740978 5762 log.cc:824] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:39.743034 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece: No bootstrap required, opened a new log
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:39.746945 5762 raft_consensus.cc:348] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } }
I20260924 10:44:39.747188 5762 raft_consensus.cc:374] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:39.747272 5762 raft_consensus.cc:729] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d30d699d5e024fb0a25417b049f81ece, State: Initialized, Role: FOLLOWER
I20260924 10:44:39.747785 5762 consensus_queue.cc:260] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } }
I20260924 10:44:39.747954 5762 raft_consensus.cc:388] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:39.748032 5762 raft_consensus.cc:482] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:39.748150 5762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:39.749188 5762 raft_consensus.cc:504] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } }
I20260924 10:44:39.749521 5762 leader_election.cc:302] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d30d699d5e024fb0a25417b049f81ece; no voters:
I20260924 10:44:39.749825 5762 leader_election.cc:288] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:39.749903 5767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:39.750089 5767 raft_consensus.cc:686] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [term 1 LEADER]: Becoming Leader. State: Replica: d30d699d5e024fb0a25417b049f81ece, State: Running, Role: LEADER
I20260924 10:44:39.750460 5767 consensus_queue.cc:237] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } }
I20260924 10:44:39.751066 5762 sys_catalog.cc:564] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:39.751595 5767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader d30d699d5e024fb0a25417b049f81ece. Latest consensus state: current_term: 1 leader_uuid: "d30d699d5e024fb0a25417b049f81ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } } }
I20260924 10:44:39.751775 5767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:39.751993 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d30d699d5e024fb0a25417b049f81ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30d699d5e024fb0a25417b049f81ece" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42569 } } }
I20260924 10:44:39.752146 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:39.752214 5773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:39.752866 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:39.738943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.2.182.62:42569)
0924 10:44:39.739347 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:39.739364 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:39.739403 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:44:39.740215 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:39.740236 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:39.740643 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:39.741017 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:39.741928 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:39.743297 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:39.744144 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:39.744830 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:39.745026 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:39.747417 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:39.747422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:39.747434 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:39.747457 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:39.749414 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:39.749875 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:39.749882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:39.749889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:39.750102 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:39.750409 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:39.750412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:39.750413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:39.750834 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:39.751037 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:39.751621 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 10:44:39.751947 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260924 10:44:39.755589 5773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:39.756070 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:39.760990 5773 catalog_manager.cc:1347] Generated new cluster ID: f487ca265e704173966121c8c74ba782
I20260924 10:44:39.761050 5773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:39.769531 5773 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:39.770331 5773 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:39.779350 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:39.774254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.2.182.62:42569)
0924 10:44:39.774361 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:39.774365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:39.774380 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:39.774421 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:39.774424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:39.774466 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:39.774552 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:39.775071 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:39.775879 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:39.777056 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:39.777413 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:39.777517 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:39.778388 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:39.778392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:39.778395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:39.778928 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:39.778959 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:39.778977 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:39.779065 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:39.779151 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:44:39.779210 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260924 10:44:39.783047 5773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d30d699d5e024fb0a25417b049f81ece: Generated new TSK 0
I20260924 10:44:39.783676 5773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:39.794109 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5694
2026-09-24T10:44:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1386 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5795](info): set up 2 sockets
Sep 24 10:44:39 dist-test-slave-6lwt krb5kdc[5795](info): commencing operation
krb5kdc: starting...
W20260924 10:44:40.792835 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:40 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:40.910065 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:34291
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:41.015442 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:41.015686 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:41.015770 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:41.019740 5808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:41.019795 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:41.019814 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:41.019860 5808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:41.019891 5808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:41.024843 5808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:34291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:41.025835 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:41.026800 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:41.034839 5813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.034869 5816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.034843 5814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:41.035130 5808 server_base.cc:1034] running on GCE node
I20260924 10:44:41.035694 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:41.036711 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:41.037904 5808 hybrid_clock.cc:648] HybridClock initialized: now 1790246681037899 us; error 58 us; skew 500 ppm
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:41.040462 5808 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:41.041303 5808 webserver.cc:460] Webserver started at http://127.2.182.62:34615/ using document root <none> and password file <none>
I20260924 10:44:41.041792 5808 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:41.041847 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:41.042008 5808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:41.043900 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "0860814c1b7743ae99a3f146db0a1a4e"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "a331d6d40b7bb02e2fdc308c289b91f3"
server_key_iv: "4db91eee4e76726b6333121aa644eb3e"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.044385 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "0860814c1b7743ae99a3f146db0a1a4e"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "a331d6d40b7bb02e2fdc308c289b91f3"
server_key_iv: "4db91eee4e76726b6333121aa644eb3e"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.047801 5808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:41.050036 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.051157 5808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:41.051286 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "0860814c1b7743ae99a3f146db0a1a4e"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "a331d6d40b7bb02e2fdc308c289b91f3"
server_key_iv: "4db91eee4e76726b6333121aa644eb3e"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.051381 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:41.075845 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:41.078583 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:41.078780 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:41.085367 5808 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:34291
I20260924 10:44:41.085384 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:34291 every 8 connection(s)
I20260924 10:44:41.086266 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:41.088873 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:41.093852 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e: Bootstrap starting.
I20260924 10:44:41.095553 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5808
I20260924 10:44:41.095655 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:41.095917 2776 external_mini_cluster.cc:1428] Setting key 891bfcfe21519a0405f61aa602b1bbd9
I20260924 10:44:41.096282 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:41.097129 5876 log.cc:824] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:41.098982 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e: No bootstrap required, opened a new log
I20260924 10:44:41.102828 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } }
I20260924 10:44:41.103111 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:41.103184 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860814c1b7743ae99a3f146db0a1a4e, State: Initialized, Role: FOLLOWER
I20260924 10:44:41.103682 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } }
I20260924 10:44:41.103852 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:41.103935 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:41.104067 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:41.104972 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } }
I20260924 10:44:41.105288 5876 leader_election.cc:302] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0860814c1b7743ae99a3f146db0a1a4e; no voters:
I20260924 10:44:41.105573 5876 leader_election.cc:288] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:41.106060 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:41.106340 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [term 1 LEADER]: Becoming Leader. State: Replica: 0860814c1b7743ae99a3f146db0a1a4e, State: Running, Role: LEADER
I20260924 10:44:41.106704 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } }
I20260924 10:44:41.107149 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:41.108127 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0860814c1b7743ae99a3f146db0a1a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } } }
I20260924 10:44:41.108294 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:41.108685 5890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:41.109401 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0860814c1b7743ae99a3f146db0a1a4e. Latest consensus state: current_term: 1 leader_uuid: "0860814c1b7743ae99a3f146db0a1a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0860814c1b7743ae99a3f146db0a1a4e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34291 } } }
I20260924 10:44:41.109490 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:41.110394 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.096998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.2.182.62:34291)
0924 10:44:41.097377 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:41.097390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:41.097421 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:44:41.098161 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:41.098179 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:41.098471 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:41.098724 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:41.099609 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.100536 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.101721 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.102159 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.102293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.105112 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:41.105120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:41.105139 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:41.105167 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:41.107138 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.107645 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.107651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.107656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.107797 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.108140 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.108143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.108144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.108594 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:41.108801 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:41.109228 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:44:41.109526 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260924 10:44:41.112093 5890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:41.117200 5890 catalog_manager.cc:1347] Generated new cluster ID: b2836bbd87ce464c816d930d289d6237
I20260924 10:44:41.117259 5890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:41.124265 5890 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:41.125389 5890 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:41.139187 5890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0860814c1b7743ae99a3f146db0a1a4e: Generated new TSK 0
I20260924 10:44:41.139889 5890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:41.171808 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:41.275450 5902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:41.275691 5902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:41.275746 5902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:41.279513 5902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:41.279561 5902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:41.279614 5902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:41.284386 5902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:41.285247 5902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:41.286187 5902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:41.294975 5910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.294970 5907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:41.295149 5902 server_base.cc:1034] running on GCE node
W20260924 10:44:41.294971 5908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:41.295680 5902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:41.296296 5902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:41.297449 5902 hybrid_clock.cc:648] HybridClock initialized: now 1790246681297431 us; error 28 us; skew 500 ppm
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:41.299966 5902 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:41.300743 5902 webserver.cc:460] Webserver started at http://127.2.182.1:46531/ using document root <none> and password file <none>
I20260924 10:44:41.301304 5902 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:41.301363 5902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:41.301558 5902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:41.303447 5902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "4345eae333304c6ca6e3683fc117e0d3"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0e2030e6f3807250f6a6d7bc45d655f5"
server_key_iv: "83a1209b6dbd3f49d2c28d6148b090dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.303983 5902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "4345eae333304c6ca6e3683fc117e0d3"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0e2030e6f3807250f6a6d7bc45d655f5"
server_key_iv: "83a1209b6dbd3f49d2c28d6148b090dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.307510 5902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:41.309887 5917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.310976 5902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:41.311098 5902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "4345eae333304c6ca6e3683fc117e0d3"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0e2030e6f3807250f6a6d7bc45d655f5"
server_key_iv: "83a1209b6dbd3f49d2c28d6148b090dd"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.311192 5902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:41.328154 5902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:41.330847 5902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:41.331043 5902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:41.331565 5902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:41.332334 5902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:41.332381 5902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.332446 5902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:41.332476 5902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.341317 5902 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:39343
I20260924 10:44:41.341346 6030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:39343 every 8 connection(s)
I20260924 10:44:41.342110 5902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:41.347038 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 5902
I20260924 10:44:41.347128 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:41.347441 2776 external_mini_cluster.cc:1428] Setting key 240a1accd9aa587adc8cfd966ffc7fdf
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:41.355767 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.343780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:44103 (local address 127.2.182.62:34291)
0924 10:44:41.343936 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:41.343941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:41.344634 (+ 693us) server_negotiation.cc:407] Connection header received
0924 10:44:41.346027 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:41.346030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:41.346074 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:41.346145 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:41.347295 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.347989 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.349319 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.349583 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.349706 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.352144 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:41.352148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:41.352151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:41.352180 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:41.353715 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.354242 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.354251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.354253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.354327 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.354722 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.354726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.354728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.354902 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:41.355030 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:41.355467 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 10:44:41.355580 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:44:41.356487 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.344055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34291 (local address 127.2.182.1:44103)
0924 10:44:41.344492 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:41.344530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:41.345850 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:41.346319 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:41.346326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:41.346649 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:41.347140 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.347171 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.348107 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.348112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.349193 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.349202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.349736 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.349740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.349915 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.350484 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:41.350503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:41.351981 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:41.353827 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.353832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.353845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.354127 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.354445 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.354449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.354451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.354622 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.355006 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:41.355013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:41.355245 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 10:44:41.355424 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 10:44:41.355715 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:41.357668 6031 heartbeater.cc:347] Connected to a master server at 127.2.182.62:34291
I20260924 10:44:41.357898 6031 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:41.358368 6031 heartbeater.cc:511] Master 127.2.182.62:34291 requested a full tablet report, sending...
I20260924 10:44:41.359797 5840 ts_manager.cc:194] Registered new tserver with Master: 4345eae333304c6ca6e3683fc117e0d3 (127.2.182.1:39343)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:44:41.361058 5840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:44103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:41.375553 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:41.481719 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:41.481971 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:41.482012 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:41.486032 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:41.486085 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:41.486184 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:41.491240 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:41.492229 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:41.493193 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:41.502202 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.502250 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.502276 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:41.502480 6036 server_base.cc:1034] running on GCE node
I20260924 10:44:41.502931 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:41.503492 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:41.504652 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790246681504636 us; error 24 us; skew 500 ppm
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:41.507373 6036 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:41.508373 6036 webserver.cc:460] Webserver started at http://127.2.182.2:34039/ using document root <none> and password file <none>
I20260924 10:44:41.508948 6036 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:41.509019 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:41.509253 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:41.511107 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "cf87fddb0b4b4bd1b23b33847893bc36"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0dbb7420cc5d42636af11abe87eff86d"
server_key_iv: "629a662bc20dd4a2a4517c2c199255c6"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.511627 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "cf87fddb0b4b4bd1b23b33847893bc36"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0dbb7420cc5d42636af11abe87eff86d"
server_key_iv: "629a662bc20dd4a2a4517c2c199255c6"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.515172 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:41.517377 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.518414 6036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:41.518540 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "cf87fddb0b4b4bd1b23b33847893bc36"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "0dbb7420cc5d42636af11abe87eff86d"
server_key_iv: "629a662bc20dd4a2a4517c2c199255c6"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.518638 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:41.547164 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:41.549955 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:41.550154 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:41.550724 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:41.551494 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:41.551570 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.551657 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:41.551712 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.560387 6036 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:38937
I20260924 10:44:41.560415 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:38937 every 8 connection(s)
I20260924 10:44:41.561345 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:41.571563 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6036
I20260924 10:44:41.571697 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:41.571981 2776 external_mini_cluster.cc:1428] Setting key 27915e0ae677684940db3094adc5d247
I20260924 10:44:41.574246 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.562908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:51717 (local address 127.2.182.62:34291)
0924 10:44:41.563048 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:41.563051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:41.563795 (+ 744us) server_negotiation.cc:407] Connection header received
0924 10:44:41.565250 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:41.565253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:41.565300 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:41.565380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:41.566440 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.566986 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.568020 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.568227 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.568305 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.570733 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:41.570736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:41.570738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:41.570760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:41.572160 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.572721 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.572724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.572726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.572811 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.573160 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.573166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.573168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.573395 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:41.573485 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:41.574036 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 10:44:41.574122 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:44:41.575385 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.563241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34291 (local address 127.2.182.2:51717)
0924 10:44:41.563645 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:41.563691 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:41.565058 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:41.565502 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:41.565509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:41.565874 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:41.566314 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.566326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.567083 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.567086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.567916 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.567924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.568347 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.568349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.568576 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.569204 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:41.569219 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:41.570580 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:41.572277 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.572282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.572296 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.572601 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.572943 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.572945 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.572946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.573068 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.573514 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:41.573520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:41.573796 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 10:44:41.573986 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 10:44:41.574328 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260924 10:44:41.576516 6165 heartbeater.cc:347] Connected to a master server at 127.2.182.62:34291
I20260924 10:44:41.576749 6165 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:41.577226 6165 heartbeater.cc:511] Master 127.2.182.62:34291 requested a full tablet report, sending...
I20260924 10:44:41.578245 5840 ts_manager.cc:194] Registered new tserver with Master: cf87fddb0b4b4bd1b23b33847893bc36 (127.2.182.2:38937)
I20260924 10:44:41.578784 5840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:51717
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:41.598237 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:41.706171 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:41.706454 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:41.706530 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:41.710685 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:41.710757 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:41.710836 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:41.716025 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:41.716984 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:41.717924 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:41.727777 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.727784 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:41.727782 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:41.728047 6170 server_base.cc:1034] running on GCE node
I20260924 10:44:41.728510 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:41.729130 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:41.730286 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790246681730269 us; error 30 us; skew 500 ppm
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:41.732805 6170 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:41.733592 6170 webserver.cc:460] Webserver started at http://127.2.182.3:42205/ using document root <none> and password file <none>
I20260924 10:44:41.734083 6170 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:41.734136 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:41.734287 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:41.736150 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "48e8cabca51743f899d3e7f8d16c87a1"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "9b09a23a42cc2b3f31c3c43506467e2a"
server_key_iv: "9ca1992f492664b9b9525125ad176578"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.736640 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "48e8cabca51743f899d3e7f8d16c87a1"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "9b09a23a42cc2b3f31c3c43506467e2a"
server_key_iv: "9ca1992f492664b9b9525125ad176578"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.740070 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:41.742266 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.743404 6170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:41.743494 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "48e8cabca51743f899d3e7f8d16c87a1"
format_stamp: "Formatted at 2026-09-24 10:44:41 on dist-test-slave-6lwt"
server_key: "9b09a23a42cc2b3f31c3c43506467e2a"
server_key_iv: "9ca1992f492664b9b9525125ad176578"
server_key_version: "encryptionkey@0"
I20260924 10:44:41.743568 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:41.756647 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:41.759686 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:41.759831 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:41.760288 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:41.761055 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:41.761101 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.761138 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:41.761159 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:41.769438 6170 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:34825
I20260924 10:44:41.769501 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:34825 every 8 connection(s)
I20260924 10:44:41.770280 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:41.773449 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6170
I20260924 10:44:41.773532 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:41.773792 2776 external_mini_cluster.cc:1428] Setting key b123881068e601151be9ee1f2c6c5400
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:41.783061 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.772010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:40609 (local address 127.2.182.62:34291)
0924 10:44:41.772144 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:41.772148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:41.772924 (+ 776us) server_negotiation.cc:407] Connection header received
0924 10:44:41.774576 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:41.774581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:41.774631 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:41.774722 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:41.775817 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.776345 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.777533 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.777751 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.777861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.780086 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:41.780089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:41.780091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:41.780113 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:41.781408 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.781843 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.781846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.781847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.781917 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.782199 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.782201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.782203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.782379 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:41.782451 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:41.782851 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:44:41.782951 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:41.783821 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.772300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34291 (local address 127.2.182.3:40609)
0924 10:44:41.772765 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:41.772804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:41.774250 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:41.774906 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:41.774913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:41.775242 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:41.775668 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.775680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.776467 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.776471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.777406 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:41.777415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.777894 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.777898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:41.778059 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.778554 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:41.778570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:41.779929 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:41.781506 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.781510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.781521 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.781755 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.781994 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:41.781996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:41.781997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:41.782122 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:41.782460 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:41.782464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:41.782663 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 10:44:41.782806 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:44:41.783045 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260924 10:44:41.784858 6299 heartbeater.cc:347] Connected to a master server at 127.2.182.62:34291
I20260924 10:44:41.785125 6299 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:41.785583 6299 heartbeater.cc:511] Master 127.2.182.62:34291 requested a full tablet report, sending...
I20260924 10:44:41.786649 5839 ts_manager.cc:194] Registered new tserver with Master: 48e8cabca51743f899d3e7f8d16c87a1 (127.2.182.3:34825)
I20260924 10:44:41.787199 5839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:40609
I20260924 10:44:41.787336 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246681, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:41.811870 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:41.803839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.2.182.62:34291)
0924 10:44:41.803970 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:41.803974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:41.804149 (+ 175us) server_negotiation.cc:407] Connection header received
0924 10:44:41.804259 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:41.804261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:41.804301 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:41.804379 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:41.805075 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.805574 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.806375 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:41.806564 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:41.806650 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:41.809009 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:41.809012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:41.809014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:41.809034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:41.810482 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.810838 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.810840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.810842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.810935 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:41.811139 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:41.811141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:41.811143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:41.811326 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:41.811425 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:41.811581 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:44:41.811665 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 10:44:41.812282 5839 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44656
I20260924 10:44:41.813037 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5902
I20260924 10:44:41.819119 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6036
I20260924 10:44:41.824268 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6170
I20260924 10:44:41.829183 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 5808
2026-09-24T10:44:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2036 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[6311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[6311](info): set up 2 sockets
Sep 24 10:44:41 dist-test-slave-6lwt krb5kdc[6311](info): commencing operation
krb5kdc: starting...
W20260924 10:44:42.837246 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:42 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:42.953701 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37101
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:43.056592 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:43.056845 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:43.056950 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:43.060699 6324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:43.060774 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:43.060812 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:43.060848 6324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:43.060882 6324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:43.065838 6324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:37101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:43.066743 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:43.067631 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:43.075712 6329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:43.075734 6332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.075923 6324 server_base.cc:1034] running on GCE node
W20260924 10:44:43.075841 6330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.076507 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:43.077577 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:43.078724 6324 hybrid_clock.cc:648] HybridClock initialized: now 1790246683078697 us; error 42 us; skew 500 ppm
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:43.081418 6324 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:43.082187 6324 webserver.cc:460] Webserver started at http://127.2.182.62:43921/ using document root <none> and password file <none>
I20260924 10:44:43.082688 6324 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:43.082747 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:43.083009 6324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:43.084941 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "38fc01db471d479eacc821f397ee5f30"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "413df44e204ed669787b9cdd57920743"
server_key_iv: "90167473cf51b785385fe3eabdf2330f"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.085482 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "38fc01db471d479eacc821f397ee5f30"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "413df44e204ed669787b9cdd57920743"
server_key_iv: "90167473cf51b785385fe3eabdf2330f"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.088927 6324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:44:43.091116 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.092270 6324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:43.092401 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "38fc01db471d479eacc821f397ee5f30"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "413df44e204ed669787b9cdd57920743"
server_key_iv: "90167473cf51b785385fe3eabdf2330f"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.092499 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:43.104080 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:43.107102 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:43.107285 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:43.113535 6324 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:37101
I20260924 10:44:43.113590 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:37101 every 8 connection(s)
I20260924 10:44:43.114398 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:43.116968 6392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:43.119297 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6324
I20260924 10:44:43.119429 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:43.119719 2776 external_mini_cluster.cc:1428] Setting key 6b17de640a64fc435251b6f77db82d69
I20260924 10:44:43.122687 6392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: Bootstrap starting.
I20260924 10:44:43.125114 6392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:43.125909 6392 log.cc:824] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:43.128091 6392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: No bootstrap required, opened a new log
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:43.132057 6392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } }
I20260924 10:44:43.132290 6392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:43.132342 6392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fc01db471d479eacc821f397ee5f30, State: Initialized, Role: FOLLOWER
I20260924 10:44:43.132807 6392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } }
I20260924 10:44:43.132972 6392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:43.133073 6392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:43.133169 6392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:43.134131 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.121060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.2.182.62:37101)
0924 10:44:43.121407 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.121421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.121452 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:44:43.122058 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.122073 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.122367 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.122622 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.123517 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.124471 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.125646 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.126148 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.126293 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.128862 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.128867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.128881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.128930 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.131190 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.131580 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.131586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.131591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.131717 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.131980 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.131983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.131984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.132422 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.132596 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.132929 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.133216 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260924 10:44:43.134153 6392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } }
I20260924 10:44:43.134469 6392 leader_election.cc:302] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fc01db471d479eacc821f397ee5f30; no voters:
I20260924 10:44:43.134776 6392 leader_election.cc:288] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:43.134879 6397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:43.135075 6397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [term 1 LEADER]: Becoming Leader. State: Replica: 38fc01db471d479eacc821f397ee5f30, State: Running, Role: LEADER
I20260924 10:44:43.135416 6397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } }
I20260924 10:44:43.135932 6392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:43.137501 6399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fc01db471d479eacc821f397ee5f30. Latest consensus state: current_term: 1 leader_uuid: "38fc01db471d479eacc821f397ee5f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } } }
I20260924 10:44:43.137542 6398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fc01db471d479eacc821f397ee5f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc01db471d479eacc821f397ee5f30" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37101 } } }
I20260924 10:44:43.137710 6399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:43.137715 6398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:43.140755 6412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:43.140828 6412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:43.140892 6410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:43.141566 6410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:43.146180 6410 catalog_manager.cc:1347] Generated new cluster ID: 1d3f4cc2e3844b558ba3cac1c836805b
I20260924 10:44:43.146234 6410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:43.163266 6410 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:43.164151 6410 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:43.174449 6410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38fc01db471d479eacc821f397ee5f30: Generated new TSK 0
I20260924 10:44:43.175109 6410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:43.211414 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:43.331354 6418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:43.331604 6418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:43.331643 6418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:43.335464 6418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:43.335511 6418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:43.335566 6418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:43.341197 6418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:43.342068 6418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:43.342988 6418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:43.352146 6424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.352210 6418 server_base.cc:1034] running on GCE node
W20260924 10:44:43.352154 6426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:43.352162 6423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.352737 6418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:43.353327 6418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:43.354480 6418 hybrid_clock.cc:648] HybridClock initialized: now 1790246683354466 us; error 25 us; skew 500 ppm
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:43.357071 6418 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:43.357857 6418 webserver.cc:460] Webserver started at http://127.2.182.1:37785/ using document root <none> and password file <none>
I20260924 10:44:43.358394 6418 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:43.358479 6418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:43.358675 6418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:43.360632 6418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "eacc169b88944aba885910f5d6aa7822"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "1ff11865847292f6ec80d24a31ebf548"
server_key_iv: "3d27424f09cccc7551b13335b39b7977"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.361186 6418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "eacc169b88944aba885910f5d6aa7822"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "1ff11865847292f6ec80d24a31ebf548"
server_key_iv: "3d27424f09cccc7551b13335b39b7977"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.364414 6418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:44:43.366622 6433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.367830 6418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:43.367951 6418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "eacc169b88944aba885910f5d6aa7822"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "1ff11865847292f6ec80d24a31ebf548"
server_key_iv: "3d27424f09cccc7551b13335b39b7977"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.368083 6418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:43.392498 6418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:43.395267 6418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:43.395450 6418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:43.395996 6418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:43.396790 6418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:43.396857 6418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.396940 6418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:43.396983 6418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.405462 6418 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36443
I20260924 10:44:43.405490 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:36443 every 8 connection(s)
I20260924 10:44:43.406253 6418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:43.407648 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6418
I20260924 10:44:43.407763 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:43.408097 2776 external_mini_cluster.cc:1428] Setting key 35db324fae58b8dcc6aaf8601bc1df62
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:44:43.420136 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.407867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:45313 (local address 127.2.182.62:37101)
0924 10:44:43.408034 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.408039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.408788 (+ 749us) server_negotiation.cc:407] Connection header received
0924 10:44:43.410185 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.410190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.410245 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.410335 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.411702 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.412199 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.413941 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.414147 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.414223 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.416833 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.416836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.416839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.416861 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.418268 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.418744 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.418748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.418750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.418828 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.419141 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.419146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.419148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.419313 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.419411 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.419867 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.419976 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:44:43.421100 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.408198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37101 (local address 127.2.182.1:45313)
0924 10:44:43.408651 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:43.408700 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:43.409984 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:43.410477 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:43.410483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:43.410910 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:43.411553 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.411572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.412323 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.412330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.413814 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.413828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.414296 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.414303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.414549 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.415121 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:43.415145 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:43.416685 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:43.418389 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.418394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.418406 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.418636 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.418918 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.418922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.418924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.419049 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.419420 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:43.419428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:43.419638 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 10:44:43.419814 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 10:44:43.420087 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260924 10:44:43.422397 6547 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37101
I20260924 10:44:43.422610 6547 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:43.423060 6547 heartbeater.cc:511] Master 127.2.182.62:37101 requested a full tablet report, sending...
I20260924 10:44:43.424412 6356 ts_manager.cc:194] Registered new tserver with Master: eacc169b88944aba885910f5d6aa7822 (127.2.182.1:36443)
I20260924 10:44:43.425526 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:45313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:43.435231 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:43.540069 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:43.540282 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:43.540320 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:43.544180 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:43.544224 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:43.544275 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:43.549332 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:43.550220 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:43.551059 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:43.559989 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.560124 6552 server_base.cc:1034] running on GCE node
W20260924 10:44:43.559999 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:43.559998 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.560704 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:43.561285 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:43.562494 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790246683562480 us; error 24 us; skew 500 ppm
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:43.565088 6552 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:43.565860 6552 webserver.cc:460] Webserver started at http://127.2.182.2:43541/ using document root <none> and password file <none>
I20260924 10:44:43.566386 6552 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:43.566473 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:43.566691 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:43.568528 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "5b23cb6a317542dd9a671a1016f5b6cd"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "d9add824ac0f95a70f8123710e4e40bf"
server_key_iv: "4b7a3c296cef0c9daab731870f8622de"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.569100 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "5b23cb6a317542dd9a671a1016f5b6cd"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "d9add824ac0f95a70f8123710e4e40bf"
server_key_iv: "4b7a3c296cef0c9daab731870f8622de"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.572738 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:44:43.575069 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.576174 6552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:43.576310 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "5b23cb6a317542dd9a671a1016f5b6cd"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "d9add824ac0f95a70f8123710e4e40bf"
server_key_iv: "4b7a3c296cef0c9daab731870f8622de"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.576412 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:43.585922 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:43.588438 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:43.588625 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:43.589216 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:43.589974 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:43.590042 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.590107 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:43.590162 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.598753 6552 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:45275
I20260924 10:44:43.598770 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:45275 every 8 connection(s)
I20260924 10:44:43.599579 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:43.600800 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6552
I20260924 10:44:43.600937 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:43.601199 2776 external_mini_cluster.cc:1428] Setting key f387f20e8625bf8d25ab095b24646a95
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
I20260924 10:44:43.614004 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.601354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:59613 (local address 127.2.182.62:37101)
0924 10:44:43.601500 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.601504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.602416 (+ 912us) server_negotiation.cc:407] Connection header received
0924 10:44:43.604500 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.604504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.604567 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.604666 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.605819 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.606365 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.607496 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.607723 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.607833 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.610670 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.610673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.610676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.610698 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.612089 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.612538 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.612540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.612542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.612611 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.612932 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.612936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.612937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.613128 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.613205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.613728 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.613820 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:44:43.614784 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.601791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37101 (local address 127.2.182.2:59613)
0924 10:44:43.602273 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:43.602312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:43.604263 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:43.604828 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:43.604835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:43.605224 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:43.605674 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.605687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.606497 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.606502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.607358 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.607367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.607867 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.607871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.608040 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.608662 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:43.608677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:43.610499 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:43.612199 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.612204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.612214 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.612454 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.612688 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.612690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.612692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.612817 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.613217 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:43.613222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:43.613431 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 10:44:43.613659 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 10:44:43.613970 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260924 10:44:43.615916 6681 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37101
I20260924 10:44:43.616155 6681 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:43.616621 6681 heartbeater.cc:511] Master 127.2.182.62:37101 requested a full tablet report, sending...
I20260924 10:44:43.617686 6356 ts_manager.cc:194] Registered new tserver with Master: 5b23cb6a317542dd9a671a1016f5b6cd (127.2.182.2:45275)
I20260924 10:44:43.618556 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:59613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:43.627830 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:43.733134 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:43.733397 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:43.733479 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:43.737689 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:43.737767 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:43.737875 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:43.744860 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:43.746229 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:43.747555 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:43.757153 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:43.757153 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:43.757166 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:43.757541 6686 server_base.cc:1034] running on GCE node
I20260924 10:44:43.757970 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:43.758569 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:43.759753 6686 hybrid_clock.cc:648] HybridClock initialized: now 1790246683759719 us; error 49 us; skew 500 ppm
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:43.762542 6686 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:43.763314 6686 webserver.cc:460] Webserver started at http://127.2.182.3:33149/ using document root <none> and password file <none>
I20260924 10:44:43.763804 6686 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:43.763860 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:43.764015 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:43.765967 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "4e4ab36fcad74a67a96a6645b3d6d544"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "27da8259dc1743874cd76f3dbb876270"
server_key_iv: "c89caaa6623291c3018aa72be3cc1280"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.766470 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "4e4ab36fcad74a67a96a6645b3d6d544"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "27da8259dc1743874cd76f3dbb876270"
server_key_iv: "c89caaa6623291c3018aa72be3cc1280"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.770061 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:43.772406 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.773679 6686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:43.773797 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "4e4ab36fcad74a67a96a6645b3d6d544"
format_stamp: "Formatted at 2026-09-24 10:44:43 on dist-test-slave-6lwt"
server_key: "27da8259dc1743874cd76f3dbb876270"
server_key_iv: "c89caaa6623291c3018aa72be3cc1280"
server_key_version: "encryptionkey@0"
I20260924 10:44:43.773897 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:43.792397 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:43.795629 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:43.795840 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:43.796451 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:43.797258 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:43.797344 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.797416 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:43.797479 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:43.806087 6686 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:37027
I20260924 10:44:43.806110 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:37027 every 8 connection(s)
I20260924 10:44:43.806905 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:43.813344 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6686
I20260924 10:44:43.813436 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:43.813699 2776 external_mini_cluster.cc:1428] Setting key 0df0a873f63d69ad66fd451791ad485a
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:43.819527 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.808591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:47395 (local address 127.2.182.62:37101)
0924 10:44:43.808724 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.808728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.809511 (+ 783us) server_negotiation.cc:407] Connection header received
0924 10:44:43.810838 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.810842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.810898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.810972 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.812041 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.812531 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.813712 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.813957 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.814081 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.816448 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.816451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.816453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.816474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.817916 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.818331 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.818334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.818335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.818401 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.818680 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.818682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.818683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.818861 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.818931 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.819333 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.819417 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:44:43.820341 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.808961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37101 (local address 127.2.182.3:47395)
0924 10:44:43.809365 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:43.809406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:43.810620 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:43.811128 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:43.811135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:43.811466 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:43.811901 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.811913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.812645 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.812649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.813604 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:43.813613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.814113 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.814117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:43.814289 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.814873 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:43.814888 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:43.816303 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:43.818006 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.818009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.818020 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.818249 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.818479 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:43.818481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:43.818482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:43.818609 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:43.818941 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:43.818945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:43.819148 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 10:44:43.819288 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:44:43.819507 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260924 10:44:43.821372 6815 heartbeater.cc:347] Connected to a master server at 127.2.182.62:37101
I20260924 10:44:43.821593 6815 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:43.822049 6815 heartbeater.cc:511] Master 127.2.182.62:37101 requested a full tablet report, sending...
I20260924 10:44:43.823014 6356 ts_manager.cc:194] Registered new tserver with Master: 4e4ab36fcad74a67a96a6645b3d6d544 (127.2.182.3:37027)
I20260924 10:44:43.823549 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:47395
I20260924 10:44:43.827308 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:43.851948 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.843031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.2.182.1:36443)
0924 10:44:43.843402 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.843412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.843434 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:44:43.843480 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.843486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.843662 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.843775 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.844421 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.845245 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.846040 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.846391 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.846468 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.848803 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.848811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.848823 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.848844 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.850315 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.850805 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.850810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.850811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.850883 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.851098 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.851100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.851101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.851310 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.851462 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.851593 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.851739 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":160,"threads_started":1}
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:43.862313 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:43.854705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.2.182.62:37101)
0924 10:44:43.854847 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:43.854853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:43.854891 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:44:43.855038 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:43.855046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:43.855087 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:43.855171 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:43.855742 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.856286 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.857108 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:43.857346 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:43.857455 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:43.859638 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:43.859641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:43.859643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:43.859663 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:43.861009 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.861344 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.861346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.861348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.861415 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:43.861623 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:43.861625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:43.861626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:43.861819 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:43.861913 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:43.862119 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 10:44:43.862202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 10:44:43.862679 6356 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60582
I20260924 10:44:43.863322 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6418
I20260924 10:44:43.868672 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6552
I20260924 10:44:43.874192 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6686
I20260924 10:44:43.879251 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6324
2026-09-24T10:44:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2050 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6827](info): set up 2 sockets
Sep 24 10:44:43 dist-test-slave-6lwt krb5kdc[6827](info): commencing operation
krb5kdc: starting...
W20260924 10:44:44.889362 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:44 dist-test-slave-6lwt krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:45.005525 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38443
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:45.111078 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:45.111310 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:45.111351 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:45.115175 6840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:45.115217 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:45.115236 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:45.115252 6840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:45.115266 6840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:45.120114 6840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:45.120967 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:45.121829 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:45.129446 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.129428 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.129427 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:45.129776 6840 server_base.cc:1034] running on GCE node
I20260924 10:44:45.130299 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:45.131261 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:45.132443 6840 hybrid_clock.cc:648] HybridClock initialized: now 1790246685132430 us; error 28 us; skew 500 ppm
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:45.142506 6840 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:45.143252 6840 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:45.143343 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:45.143573 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:45.145576 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "415e7fbba4834299ae890024d7907d42"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "df14175718656efcc151974db09dc6a5"
server_key_iv: "243e5432b32b1741d420a94b051ba48a"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.146116 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "415e7fbba4834299ae890024d7907d42"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "df14175718656efcc151974db09dc6a5"
server_key_iv: "243e5432b32b1741d420a94b051ba48a"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.149546 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:44:45.152139 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.153726 6840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:44:45.153855 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "415e7fbba4834299ae890024d7907d42"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "df14175718656efcc151974db09dc6a5"
server_key_iv: "243e5432b32b1741d420a94b051ba48a"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.153961 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:45.174527 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:45.177300 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:45.177439 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:45.184036 6840 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38443
I20260924 10:44:45.184347 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:38443 every 8 connection(s)
I20260924 10:44:45.184705 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:45.187299 6907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:45.190831 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6840
I20260924 10:44:45.190937 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:45.191221 2776 external_mini_cluster.cc:1428] Setting key f53e3d7d324f44d6eb7bbd679ab7ec8f
I20260924 10:44:45.193001 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: Bootstrap starting.
I20260924 10:44:45.195487 6907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:45.196302 6907 log.cc:824] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:45.198124 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: No bootstrap required, opened a new log
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:45.202184 6907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } }
I20260924 10:44:45.202364 6907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:45.202401 6907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415e7fbba4834299ae890024d7907d42, State: Initialized, Role: FOLLOWER
I20260924 10:44:45.202836 6907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } }
I20260924 10:44:45.203012 6907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:45.203078 6907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:45.203183 6907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:45.203956 6907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } }
I20260924 10:44:45.204344 6907 leader_election.cc:302] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415e7fbba4834299ae890024d7907d42; no voters:
I20260924 10:44:45.204676 6907 leader_election.cc:288] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:45.204749 6912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:45.204936 6912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [term 1 LEADER]: Becoming Leader. State: Replica: 415e7fbba4834299ae890024d7907d42, State: Running, Role: LEADER
I20260924 10:44:45.205299 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.192521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.2.182.62:38443)
0924 10:44:45.192905 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.192924 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.192964 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:44:45.193723 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.193744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.194135 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.194476 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:45.195125 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.196558 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.197474 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.198069 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.198255 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.200761 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:45.200766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:45.200776 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:45.200799 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:45.202322 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.202760 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.202772 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.202776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.202874 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.203206 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.203212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.203214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.203588 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:45.203795 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.204078 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.204348 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260924 10:44:45.205276 6912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } }
I20260924 10:44:45.205866 6907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:45.206399 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415e7fbba4834299ae890024d7907d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } } }
I20260924 10:44:45.206511 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:45.207121 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415e7fbba4834299ae890024d7907d42. Latest consensus state: current_term: 1 leader_uuid: "415e7fbba4834299ae890024d7907d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e7fbba4834299ae890024d7907d42" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38443 } } }
I20260924 10:44:45.207212 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:45.210362 6928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:45.210448 6928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:45.210521 6918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:45.211169 6918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:45.215694 6918 catalog_manager.cc:1347] Generated new cluster ID: bd46b706160549558a6c861c22a7c660
I20260924 10:44:45.215752 6918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:45.247272 6918 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:45.248325 6918 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:45.260812 6918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: Generated new TSK 0
I20260924 10:44:45.261451 6918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:45.295886 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:45.399933 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:45.400210 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:45.400255 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:45.404143 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:45.404194 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:45.404249 6933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:45.409458 6933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:45.410343 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:45.411229 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:45.419979 6939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.420019 6938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:45.420099 6933 server_base.cc:1034] running on GCE node
W20260924 10:44:45.420039 6941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:45.420583 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:45.421240 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:45.422443 6933 hybrid_clock.cc:648] HybridClock initialized: now 1790246685422430 us; error 26 us; skew 500 ppm
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:45.425319 6933 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:45.426033 6933 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:45.426091 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:45.426285 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:45.428150 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "02996ce35d544cb382c5c9daa7faf148"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "98437e0c60ce28794022f35a6542ac24"
server_key_iv: "a469982614ea0587cffe5a756d6a7110"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.428682 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "02996ce35d544cb382c5c9daa7faf148"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "98437e0c60ce28794022f35a6542ac24"
server_key_iv: "a469982614ea0587cffe5a756d6a7110"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.432420 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:45.434991 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.436185 6933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:44:45.436267 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "02996ce35d544cb382c5c9daa7faf148"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "98437e0c60ce28794022f35a6542ac24"
server_key_iv: "a469982614ea0587cffe5a756d6a7110"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.436337 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:45.451006 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:45.453475 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:45.453600 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:45.454041 6933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:45.454720 6933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:45.454772 6933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.454808 6933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:45.454830 6933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.463387 6933 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:43995
I20260924 10:44:45.463403 7060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:43995 every 8 connection(s)
I20260924 10:44:45.463958 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:45.471369 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 6933
I20260924 10:44:45.471489 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:45.471825 2776 external_mini_cluster.cc:1428] Setting key b26954264ae402536a08d9704f68860e
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:45.478449 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.465530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:46965 (local address 127.2.182.62:38443)
0924 10:44:45.465651 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.465655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.466419 (+ 764us) server_negotiation.cc:407] Connection header received
0924 10:44:45.467700 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.467703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.467772 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.467871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:45.468952 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.469473 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.470587 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.470978 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.471106 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.474072 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:45.474077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:45.474081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:45.474113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:45.475976 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.476577 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.476580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.476582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.476678 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.477122 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.477128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.477131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.477373 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:45.477498 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.478135 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.478270 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:45.479590 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.465861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38443 (local address 127.2.182.1:46965)
0924 10:44:45.466284 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.466322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.467502 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.468029 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.468035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.468371 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:45.468787 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.468798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.469593 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.469600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.470459 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.470468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.471232 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.471237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.471457 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.472098 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:45.472119 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:45.473925 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:45.476076 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.476084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.476100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.476460 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.476790 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.476795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.476800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.477007 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.477511 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:45.477519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:45.477820 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.478062 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.478441 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260924 10:44:45.480959 7061 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38443
I20260924 10:44:45.481185 7061 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:45.481637 7061 heartbeater.cc:511] Master 127.2.182.62:38443 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:45.483107 6871 ts_manager.cc:194] Registered new tserver with Master: 02996ce35d544cb382c5c9daa7faf148 (127.2.182.1:43995)
I20260924 10:44:45.484261 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:46965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:45.498075 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:45.602213 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:45.602491 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:45.602571 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:45.606568 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:45.606647 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:45.606748 7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:45.611799 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:45.612746 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:45.613715 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:45.622641 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.622680 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.622680 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:45.622835 7066 server_base.cc:1034] running on GCE node
I20260924 10:44:45.623327 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:45.623916 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:45.625099 7066 hybrid_clock.cc:648] HybridClock initialized: now 1790246685625071 us; error 43 us; skew 500 ppm
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:45.627787 7066 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:45.628513 7066 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:45.628604 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:45.628822 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:45.630743 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "1d5d5fc74a74431399c0bc2595cac57f"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "06abbbea8823628b3a0cd45086b2aa77"
server_key_iv: "0e5e2b7c9a757fc4c41eb5d5e56e37b9"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.631284 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "1d5d5fc74a74431399c0bc2595cac57f"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "06abbbea8823628b3a0cd45086b2aa77"
server_key_iv: "0e5e2b7c9a757fc4c41eb5d5e56e37b9"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.634757 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:45.636992 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.638118 7066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:45.638284 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "1d5d5fc74a74431399c0bc2595cac57f"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "06abbbea8823628b3a0cd45086b2aa77"
server_key_iv: "0e5e2b7c9a757fc4c41eb5d5e56e37b9"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.638394 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:45.669791 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:45.673126 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:45.673327 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:45.673982 7066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:45.674809 7066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:45.674887 7066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.674976 7066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:45.675028 7066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.685130 7066 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:40365
I20260924 10:44:45.685199 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:40365 every 8 connection(s)
I20260924 10:44:45.685721 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:45.699327 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7066
I20260924 10:44:45.699443 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:45.699728 2776 external_mini_cluster.cc:1428] Setting key 2c8191c0a20948a11026fe7aac98805d
I20260924 10:44:45.704408 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.687681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33541 (local address 127.2.182.62:38443)
0924 10:44:45.687837 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.687844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.688726 (+ 882us) server_negotiation.cc:407] Connection header received
0924 10:44:45.690570 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.690573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.690618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.690689 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:45.692314 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.693145 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.694959 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.695370 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.695454 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.698798 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:45.698803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:45.698808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:45.698842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:45.701711 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.702361 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.702367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.702369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.702465 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.702952 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.702958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.702960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.703220 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:45.703309 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.704016 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.704158 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:45.706147 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.688044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38443 (local address 127.2.182.2:33541)
0924 10:44:45.688544 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.688602 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.690347 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.690876 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.690887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.691435 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:45.692133 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.692152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.693330 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.693334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.694811 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.694824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.695661 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.695667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.695875 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.696532 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:45.696557 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:45.698586 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:45.701837 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.701845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.701865 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.702246 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.702590 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.702596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.702601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.702812 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.703398 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:45.703404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:45.703720 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.704609 (+ 889us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.704989 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260924 10:44:45.707558 7194 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38443
I20260924 10:44:45.707806 7194 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:45.708355 7194 heartbeater.cc:511] Master 127.2.182.62:38443 requested a full tablet report, sending...
I20260924 10:44:45.709440 6871 ts_manager.cc:194] Registered new tserver with Master: 1d5d5fc74a74431399c0bc2595cac57f (127.2.182.2:40365)
I20260924 10:44:45.710237 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:33541
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:45.726249 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:45.832211 7199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:45.832455 7199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:45.832525 7199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:45.836344 7199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:45.836418 7199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:45.836504 7199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:45.841674 7199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:45.842597 7199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:45.843519 7199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:45.852222 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.852198 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:45.852262 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:45.852442 7199 server_base.cc:1034] running on GCE node
I20260924 10:44:45.852981 7199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:45.853540 7199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:45.854691 7199 hybrid_clock.cc:648] HybridClock initialized: now 1790246685854675 us; error 27 us; skew 500 ppm
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:45.857280 7199 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:45.857916 7199 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:45.857975 7199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:45.858121 7199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:45.859992 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "40d9823ab4994dfca0c0d2f56f0e149b"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "2fc6610298428e448a6082386960b2e2"
server_key_iv: "d2a34e104514b2a0c4c17326061d0f1e"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.860438 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "40d9823ab4994dfca0c0d2f56f0e149b"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "2fc6610298428e448a6082386960b2e2"
server_key_iv: "d2a34e104514b2a0c4c17326061d0f1e"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.863678 7199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:45.865846 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.866923 7199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:45.867007 7199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "40d9823ab4994dfca0c0d2f56f0e149b"
format_stamp: "Formatted at 2026-09-24 10:44:45 on dist-test-slave-6lwt"
server_key: "2fc6610298428e448a6082386960b2e2"
server_key_iv: "d2a34e104514b2a0c4c17326061d0f1e"
server_key_version: "encryptionkey@0"
I20260924 10:44:45.867079 7199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:45.881830 7199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:45.884434 7199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:45.884586 7199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:45.885141 7199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:45.885856 7199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:45.885922 7199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.885959 7199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:45.885977 7199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:45.895568 7199 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:43697
I20260924 10:44:45.895599 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:43697 every 8 connection(s)
I20260924 10:44:45.896248 7199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:45.901685 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7199
I20260924 10:44:45.901805 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:45.902076 2776 external_mini_cluster.cc:1428] Setting key 05ec4b28b268a46ea04aa812434a98c8
Sep 24 10:44:45 dist-test-slave-6lwt krb5kdc[6827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:45.909801 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.897988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:55893 (local address 127.2.182.62:38443)
0924 10:44:45.898153 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.898158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.899063 (+ 905us) server_negotiation.cc:407] Connection header received
0924 10:44:45.900331 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.900334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.900379 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.900454 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:45.901719 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.902362 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.903796 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.904122 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.904200 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.906680 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:45.906683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:45.906685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:45.906708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:45.908088 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.908550 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.908553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.908554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.908622 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.908933 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.908942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.908943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.909115 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:45.909189 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.909604 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.909689 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:44:45.910538 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.898457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38443 (local address 127.2.182.3:55893)
0924 10:44:45.898904 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.898947 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.900142 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.900576 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.900582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.901033 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:45.901577 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.901597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.902518 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.902521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.903688 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.903700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.904272 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.904276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.904429 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.905021 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:45.905036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:45.906533 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:45.908204 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.908208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.908219 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.908467 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.908697 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:45.908699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:45.908700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:45.908819 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:45.909200 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:45.909205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:45.909415 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.909561 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.909780 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
I20260924 10:44:45.911559 7327 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38443
I20260924 10:44:45.911777 7327 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:45.912171 7327 heartbeater.cc:511] Master 127.2.182.62:38443 requested a full tablet report, sending...
I20260924 10:44:45.913125 6871 ts_manager.cc:194] Registered new tserver with Master: 40d9823ab4994dfca0c0d2f56f0e149b (127.2.182.3:43697)
I20260924 10:44:45.913643 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:55893
I20260924 10:44:45.915832 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:44:45.924882 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.917627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.2.182.62:38443)
0924 10:44:45.917758 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.917762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.917950 (+ 188us) server_negotiation.cc:407] Connection header received
0924 10:44:45.918093 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.918096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.918141 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.918221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:45.918923 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.919388 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.920188 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.920423 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.920530 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.921888 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:45.921892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:45.921902 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:45.921931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:45.923239 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.923658 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.923661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.923663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.923743 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:45.924013 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:45.924018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:45.924020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:45.924226 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:45.924365 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.924601 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.924727 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:44:45.927393 6871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:45.930459 6871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:45.944430 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.938551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:40365 (local address 127.0.0.1:55144)
0924 10:44:45.938854 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.938903 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.939176 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.940216 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.940221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.940244 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:45.940372 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.940379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.941636 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.941642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.942864 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.942874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.943907 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.943911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.944013 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.944047 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.944161 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.944278 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260924 10:44:45.944432 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.938683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.2.182.2:40365)
0924 10:44:45.939755 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.939767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.939784 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:44:45.939853 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.939858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.940057 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.940212 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:45.940529 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.941505 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.943017 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.943786 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.943917 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.944062 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.944189 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.944277 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":117,"threads_started":1}
I20260924 10:44:45.944908 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.939337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:43697 (local address 127.0.0.1:48924)
0924 10:44:45.939894 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.939909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.939990 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.940704 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.940707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.940725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:45.940810 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.940815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.942305 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.942310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.943473 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.943484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.944492 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.944495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.944583 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.944604 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.944652 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.944739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260924 10:44:45.945066 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.938694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43995 (local address 127.0.0.1:49648)
0924 10:44:45.939405 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:45.939421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:45.939510 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:45.940393 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:45.940396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:45.940410 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:45.940544 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.940551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.942150 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.942153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.943261 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:45.943268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.944472 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.944477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:45.944764 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.944784 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:44:45.944848 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:44:45.944938 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":54,"thread_start_us":95,"threads_started":1}
I20260924 10:44:45.945461 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.939534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.2.182.3:43697)
0924 10:44:45.940244 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.940255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.940275 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:44:45.940343 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.940348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.940547 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.940971 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:45.941109 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.942157 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.943606 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.944371 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.945067 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.945208 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.945252 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.945322 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":91,"threads_started":1}
I20260924 10:44:45.945464 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:45.939344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.2.182.1:43995)
0924 10:44:45.939747 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:45.939755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:45.939779 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:44:45.939853 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:45.939858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:45.940057 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:45.940218 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:45.940675 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.941961 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.943377 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:45.944152 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:45.944273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:45.944406 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:45.945217 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 10:44:45.945312 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":101,"threads_started":1}
I20260924 10:44:45.947729 6995 tablet_service.cc:1467] Processing CreateTablet for tablet beeb18cd8be744ffba4da5712e5731e7 (DEFAULT_TABLE table=test-table [id=35e22569948c4d7abbf55c852605a3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:45.947826 7261 tablet_service.cc:1467] Processing CreateTablet for tablet beeb18cd8be744ffba4da5712e5731e7 (DEFAULT_TABLE table=test-table [id=35e22569948c4d7abbf55c852605a3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:45.947762 7128 tablet_service.cc:1467] Processing CreateTablet for tablet beeb18cd8be744ffba4da5712e5731e7 (DEFAULT_TABLE table=test-table [id=35e22569948c4d7abbf55c852605a3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:45.948757 7128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeb18cd8be744ffba4da5712e5731e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:45.948763 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeb18cd8be744ffba4da5712e5731e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:45.948762 6995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeb18cd8be744ffba4da5712e5731e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:45.954583 7344 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: Bootstrap starting.
I20260924 10:44:45.955912 7345 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Bootstrap starting.
I20260924 10:44:45.956992 7344 tablet_bootstrap.cc:654] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:45.957224 7346 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Bootstrap starting.
I20260924 10:44:45.958353 7344 log.cc:824] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:45.959331 7346 tablet_bootstrap.cc:654] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:45.959862 7345 tablet_bootstrap.cc:654] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:45.960274 7344 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: No bootstrap required, opened a new log
I20260924 10:44:45.960472 7344 ts_tablet_manager.cc:1397] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:45.960657 7346 log.cc:824] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:45.961118 7345 log.cc:824] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:45.962519 7346 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: No bootstrap required, opened a new log
I20260924 10:44:45.962735 7346 ts_tablet_manager.cc:1397] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:45.963003 7345 tablet_bootstrap.cc:492] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: No bootstrap required, opened a new log
I20260924 10:44:45.963161 7345 ts_tablet_manager.cc:1397] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:44:45.964089 7344 raft_consensus.cc:348] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.964313 7344 raft_consensus.cc:374] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:45.964363 7344 raft_consensus.cc:729] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02996ce35d544cb382c5c9daa7faf148, State: Initialized, Role: FOLLOWER
I20260924 10:44:45.964823 7344 consensus_queue.cc:260] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.965420 7061 heartbeater.cc:503] Master 127.2.182.62:38443 was elected leader, sending a full tablet report...
I20260924 10:44:45.965785 7344 ts_tablet_manager.cc:1428] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:44:45.966506 7346 raft_consensus.cc:348] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.966756 7346 raft_consensus.cc:374] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:45.966833 7346 raft_consensus.cc:729] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d9823ab4994dfca0c0d2f56f0e149b, State: Initialized, Role: FOLLOWER
I20260924 10:44:45.966769 7345 raft_consensus.cc:348] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.967006 7345 raft_consensus.cc:374] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:45.967063 7345 raft_consensus.cc:729] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5d5fc74a74431399c0bc2595cac57f, State: Initialized, Role: FOLLOWER
I20260924 10:44:45.967325 7346 consensus_queue.cc:260] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.967412 7345 consensus_queue.cc:260] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:45.967976 7327 heartbeater.cc:503] Master 127.2.182.62:38443 was elected leader, sending a full tablet report...
I20260924 10:44:45.968236 7345 ts_tablet_manager.cc:1428] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:44:45.968250 7346 ts_tablet_manager.cc:1428] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:44:45.968505 7194 heartbeater.cc:503] Master 127.2.182.62:38443 was elected leader, sending a full tablet report...
I20260924 10:44:46.017931 7351 raft_consensus.cc:482] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:46.018115 7351 raft_consensus.cc:504] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:46.019207 7351 leader_election.cc:288] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d5d5fc74a74431399c0bc2595cac57f (127.2.182.2:40365), 02996ce35d544cb382c5c9daa7faf148 (127.2.182.1:43995)
I20260924 10:44:46.023103 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.019718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:60331 (local address 127.2.182.1:43995)
0924 10:44:46.019842 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.019845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.019945 (+ 100us) server_negotiation.cc:407] Connection header received
0924 10:44:46.020052 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.020055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.020102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.020186 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:46.020465 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.020989 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022137 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.022546 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022655 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:46.022718 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.022893 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.022975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:46.023106 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.019413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:40365 (local address 127.2.182.3:54235)
0924 10:44:46.019597 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.019612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.019697 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.019993 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.020012 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.020034 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:46.020162 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:46.020171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.021141 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.021143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:46.022024 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:46.022033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.022684 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:46.022807 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:46.022823 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.022918 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.022991 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":81}
I20260924 10:44:46.023123 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.019441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43995 (local address 127.2.182.3:60331)
0924 10:44:46.019840 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.019853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.019957 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.020206 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.020208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.020242 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:46.020352 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:46.020358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.021094 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.021100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:46.022017 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:46.022026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.022668 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:46.022768 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:46.022795 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.022855 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.022955 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":31,"thread_start_us":110,"threads_started":1}
I20260924 10:44:46.023140 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.019553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:54235 (local address 127.2.182.2:40365)
0924 10:44:46.019677 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.019683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.019703 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:44:46.019812 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.019814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.019875 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.019963 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:46.020275 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.021011 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022176 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.022554 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.022672 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:46.022730 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.022968 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.023037 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:44:46.023765 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeb18cd8be744ffba4da5712e5731e7" candidate_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5d5fc74a74431399c0bc2595cac57f" is_pre_election: true
I20260924 10:44:46.023896 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeb18cd8be744ffba4da5712e5731e7" candidate_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02996ce35d544cb382c5c9daa7faf148" is_pre_election: true
I20260924 10:44:46.024073 7148 raft_consensus.cc:2454] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d9823ab4994dfca0c0d2f56f0e149b in term 0.
I20260924 10:44:46.024224 7015 raft_consensus.cc:2454] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d9823ab4994dfca0c0d2f56f0e149b in term 0.
I20260924 10:44:46.024617 7214 leader_election.cc:302] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5d5fc74a74431399c0bc2595cac57f, 40d9823ab4994dfca0c0d2f56f0e149b; no voters:
I20260924 10:44:46.024946 7351 raft_consensus.cc:2781] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:46.025032 7351 raft_consensus.cc:482] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:46.025089 7351 raft_consensus.cc:3037] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.026091 7351 raft_consensus.cc:504] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:46.026443 7351 leader_election.cc:288] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [CANDIDATE]: Term 1 election: Requested vote from peers 1d5d5fc74a74431399c0bc2595cac57f (127.2.182.2:40365), 02996ce35d544cb382c5c9daa7faf148 (127.2.182.1:43995)
I20260924 10:44:46.026731 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeb18cd8be744ffba4da5712e5731e7" candidate_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5d5fc74a74431399c0bc2595cac57f"
I20260924 10:44:46.026742 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeb18cd8be744ffba4da5712e5731e7" candidate_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02996ce35d544cb382c5c9daa7faf148"
I20260924 10:44:46.026853 7015 raft_consensus.cc:3037] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.026853 7148 raft_consensus.cc:3037] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.028105 7148 raft_consensus.cc:2454] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d9823ab4994dfca0c0d2f56f0e149b in term 1.
I20260924 10:44:46.028105 7015 raft_consensus.cc:2454] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d9823ab4994dfca0c0d2f56f0e149b in term 1.
I20260924 10:44:46.028405 7216 leader_election.cc:302] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02996ce35d544cb382c5c9daa7faf148, 1d5d5fc74a74431399c0bc2595cac57f, 40d9823ab4994dfca0c0d2f56f0e149b; no voters:
I20260924 10:44:46.028548 7351 raft_consensus.cc:2781] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:46.028782 7351 raft_consensus.cc:686] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 1 LEADER]: Becoming Leader. State: Replica: 40d9823ab4994dfca0c0d2f56f0e149b, State: Running, Role: LEADER
I20260924 10:44:46.029124 7351 consensus_queue.cc:237] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } }
I20260924 10:44:46.032143 6868 catalog_manager.cc:5495] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d9823ab4994dfca0c0d2f56f0e149b (127.2.182.3). New cstate: current_term: 1 leader_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d9823ab4994dfca0c0d2f56f0e149b" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43697 } health_report { overall_health: HEALTHY } } }
I20260924 10:44:46.044064 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.040676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.2.182.3:43697)
0924 10:44:46.040788 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.040792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.040828 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:44:46.041023 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.041026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.041069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.041147 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:46.041379 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.041905 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.042836 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:46.043043 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:46.043117 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:46.043355 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:46.043714 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:46.043803 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.043853 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.043895 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:46.049412 7015 raft_consensus.cc:1261] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 1 FOLLOWER]: Refusing update from remote peer 40d9823ab4994dfca0c0d2f56f0e149b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:46.049412 7148 raft_consensus.cc:1261] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 1 FOLLOWER]: Refusing update from remote peer 40d9823ab4994dfca0c0d2f56f0e149b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:46.050100 7354 consensus_queue.cc:1035] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5d5fc74a74431399c0bc2595cac57f" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:46.050254 7351 consensus_queue.cc:1035] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [LEADER]: Connected to new peer: Peer: permanent_uuid: "02996ce35d544cb382c5c9daa7faf148" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:46.057687 7358 mvcc.cc:205] Tried to move back new op lower bound from 7332850426051641344 to 7332850425978101760. Current Snapshot: MvccSnapshot[applied={T|T < 7332850426051641344}]
I20260924 10:44:46.067340 6870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:46.067526 6870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:46.069882 6870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 415e7fbba4834299ae890024d7907d42: Sending DeleteTablet for 3 replicas of tablet beeb18cd8be744ffba4da5712e5731e7
I20260924 10:44:46.070699 6995 tablet_service.cc:1514] Processing DeleteTablet for tablet beeb18cd8be744ffba4da5712e5731e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:46 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:49648
I20260924 10:44:46.070688 7128 tablet_service.cc:1514] Processing DeleteTablet for tablet beeb18cd8be744ffba4da5712e5731e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:46 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:55144
I20260924 10:44:46.071189 7367 tablet_replica.cc:330] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: stopping tablet replica
I20260924 10:44:46.071324 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet beeb18cd8be744ffba4da5712e5731e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:46 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:48924
I20260924 10:44:46.071587 7367 raft_consensus.cc:2229] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:46.071810 7368 tablet_replica.cc:330] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148: stopping tablet replica
I20260924 10:44:46.071892 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6933
I20260924 10:44:46.072033 7367 raft_consensus.cc:2258] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:46.072093 7368 raft_consensus.cc:2229] T beeb18cd8be744ffba4da5712e5731e7 P 02996ce35d544cb382c5c9daa7faf148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:46.072499 7369 tablet_replica.cc:330] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: stopping tablet replica
I20260924 10:44:46.072748 7369 raft_consensus.cc:2229] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:46.073127 7369 raft_consensus.cc:2258] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:46.074036 7367 ts_tablet_manager.cc:1905] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:46.077463 7367 ts_tablet_manager.cc:1918] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:46.077615 7367 log.cc:1196] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/beeb18cd8be744ffba4da5712e5731e7
I20260924 10:44:46.077906 7369 ts_tablet_manager.cc:1905] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:46.078111 7367 ts_tablet_manager.cc:1939] T beeb18cd8be744ffba4da5712e5731e7 P 1d5d5fc74a74431399c0bc2595cac57f: Deleting consensus metadata
I20260924 10:44:46.079347 6855 catalog_manager.cc:4841] TS 1d5d5fc74a74431399c0bc2595cac57f (127.2.182.2:40365): tablet beeb18cd8be744ffba4da5712e5731e7 (table test-table [id=35e22569948c4d7abbf55c852605a3cf]) successfully deleted
W20260924 10:44:46.079838 6857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:43995 (error 108)
I20260924 10:44:46.079988 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7066
I20260924 10:44:46.081662 7369 ts_tablet_manager.cc:1918] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:46.081738 7369 log.cc:1196] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/beeb18cd8be744ffba4da5712e5731e7
I20260924 10:44:46.082031 7369 ts_tablet_manager.cc:1939] T beeb18cd8be744ffba4da5712e5731e7 P 40d9823ab4994dfca0c0d2f56f0e149b: Deleting consensus metadata
I20260924 10:44:46.082974 6857 catalog_manager.cc:4841] TS 40d9823ab4994dfca0c0d2f56f0e149b (127.2.182.3:43697): tablet beeb18cd8be744ffba4da5712e5731e7 (table test-table [id=35e22569948c4d7abbf55c852605a3cf]) successfully deleted
I20260924 10:44:46.083580 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.083298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43995 (local address 127.0.0.1:49662)
0924 10:44:46.083431 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.083502 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:44:46.083739 6857 catalog_manager.cc:4584] TS 02996ce35d544cb382c5c9daa7faf148 (127.2.182.1:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet beeb18cd8be744ffba4da5712e5731e7: Network error: Client connection negotiation failed: client connection to 127.2.182.1:43995: connect: Connection refused (error 111)
I20260924 10:44:46.085418 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7199
I20260924 10:44:46.090094 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 6840
2026-09-24T10:44:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2209 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T10:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:46Z Disabled control of system clock
I20260924 10:44:46.118314 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44427
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:44427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:44:46.223124 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:46.223397 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:46.223491 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:46.227559 7374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:46.227619 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:46.227638 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:46.227690 7374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:46.227726 7374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:46.232249 7374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:44427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:46.233248 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:46.234562 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:46.242760 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.242774 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.242770 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.243269 7374 server_base.cc:1034] running on GCE node
I20260924 10:44:46.243858 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:46.245057 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:46.246260 7374 hybrid_clock.cc:648] HybridClock initialized: now 1790246686246244 us; error 35 us; skew 500 ppm
I20260924 10:44:46.248153 7374 webserver.cc:460] Webserver started at http://127.2.182.62:39439/ using document root <none> and password file <none>
I20260924 10:44:46.248751 7374 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:46.248847 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:46.249081 7374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:46.251111 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "4b4d7b97f05b43c7bc70bd805c98823d"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "6b72b2165ca724b99bf78030b8b10fb6"
server_key_iv: "447b59300ea06350b2e4b5536528740a"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.251693 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "4b4d7b97f05b43c7bc70bd805c98823d"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "6b72b2165ca724b99bf78030b8b10fb6"
server_key_iv: "447b59300ea06350b2e4b5536528740a"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.255416 7374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:46.257915 7388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.259071 7374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:46.259213 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4b4d7b97f05b43c7bc70bd805c98823d"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "6b72b2165ca724b99bf78030b8b10fb6"
server_key_iv: "447b59300ea06350b2e4b5536528740a"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.259326 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:46.263872 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:46.264499 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:46.264703 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:46.272074 7374 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:44427
I20260924 10:44:46.272075 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:44427 every 8 connection(s)
I20260924 10:44:46.273041 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:46.273334 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7374
I20260924 10:44:46.273430 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:46.273782 2776 external_mini_cluster.cc:1428] Setting key 4158983c768d0e93b1ddaa1a929b259c
I20260924 10:44:46.276329 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:46.280371 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.275232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.2.182.62:44427)
0924 10:44:46.276354 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.276375 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.276421 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:44:46.277358 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.277381 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.277840 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.278203 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.278935 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.278941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.278956 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.278969 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.279032 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.279054 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.279211 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.279280 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.279509 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":533,"threads_started":1}
I20260924 10:44:46.283255 7441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: Bootstrap starting.
I20260924 10:44:46.285542 7441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:46.286226 7441 log.cc:824] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:46.288118 7441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: No bootstrap required, opened a new log
I20260924 10:44:46.291621 7441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } }
I20260924 10:44:46.291836 7441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:46.291873 7441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4d7b97f05b43c7bc70bd805c98823d, State: Initialized, Role: FOLLOWER
I20260924 10:44:46.292240 7441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } }
I20260924 10:44:46.292366 7441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:46.292415 7441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:46.292488 7441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.293552 7441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } }
I20260924 10:44:46.293840 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4d7b97f05b43c7bc70bd805c98823d; no voters:
I20260924 10:44:46.294090 7441 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:46.294199 7446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:46.294438 7446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [term 1 LEADER]: Becoming Leader. State: Replica: 4b4d7b97f05b43c7bc70bd805c98823d, State: Running, Role: LEADER
I20260924 10:44:46.294740 7446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } }
I20260924 10:44:46.295315 7441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:46.296312 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } } }
I20260924 10:44:46.296460 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:46.296759 7448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4d7b97f05b43c7bc70bd805c98823d. Latest consensus state: current_term: 1 leader_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4d7b97f05b43c7bc70bd805c98823d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44427 } } }
I20260924 10:44:46.296845 7455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:46.296862 7448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:46.299971 7455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:46.305514 7455 catalog_manager.cc:1347] Generated new cluster ID: 5d10835004f14c1484e3ae4dfe7bd190
I20260924 10:44:46.305583 7455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:46.326833 7455 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:46.327111 7455 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:46.337461 7455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: Generated new TSK 0
I20260924 10:44:46.338094 7455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:46.346863 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--builtin_ntp_servers=127.2.182.20:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:44:46.452826 7465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:46.453069 7465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:46.453106 7465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:46.457003 7465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:46.457047 7465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:46.457147 7465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:46.461714 7465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:46.462750 7465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:46.464159 7465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:46.473258 7470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.473344 7471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.473466 7465 server_base.cc:1034] running on GCE node
W20260924 10:44:46.473272 7473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.474031 7465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:46.474577 7465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:46.475741 7465 hybrid_clock.cc:648] HybridClock initialized: now 1790246686475708 us; error 60 us; skew 500 ppm
I20260924 10:44:46.477629 7465 webserver.cc:460] Webserver started at http://127.2.182.1:45481/ using document root <none> and password file <none>
I20260924 10:44:46.478185 7465 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:46.478255 7465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:46.478463 7465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:46.480468 7465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "01ca8d57260a4f46ab257a14b7b29f3b"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "68943e928c400929233771033cac5388"
server_key_iv: "15d3578ec7451e15272e3de68574be74"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.481104 7465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "01ca8d57260a4f46ab257a14b7b29f3b"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "68943e928c400929233771033cac5388"
server_key_iv: "15d3578ec7451e15272e3de68574be74"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.485100 7465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:44:46.487528 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.488750 7465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:46.488883 7465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "01ca8d57260a4f46ab257a14b7b29f3b"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "68943e928c400929233771033cac5388"
server_key_iv: "15d3578ec7451e15272e3de68574be74"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.489063 7465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:46.493664 7465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:46.494243 7465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:46.494415 7465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:46.494961 7465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:46.495905 7465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:46.495973 7465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.496047 7465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:46.496096 7465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.504663 7465 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:38175
I20260924 10:44:46.504709 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:38175 every 8 connection(s)
I20260924 10:44:46.505599 7465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:46.511602 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.507432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:58495 (local address 127.2.182.62:44427)
0924 10:44:46.507579 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.507584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.508668 (+ 1084us) server_negotiation.cc:407] Connection header received
0924 10:44:46.509981 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.509984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.510019 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.510093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.511204 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.511207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.511209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.511219 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.511260 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.511268 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.511346 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.511462 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.511498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:44:46.512424 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.507858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44427 (local address 127.2.182.1:58495)
0924 10:44:46.508530 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.508568 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.509801 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.510266 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.510272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.510358 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.510968 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.510969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.511009 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.511071 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.511346 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.511352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.511356 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.511480 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.511655 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260924 10:44:46.513054 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7465
I20260924 10:44:46.513177 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:46.513517 2776 external_mini_cluster.cc:1428] Setting key 42be14b8a66a2303091d5b29168679a2
I20260924 10:44:46.513787 7593 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44427
I20260924 10:44:46.514019 7593 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:46.514456 7593 heartbeater.cc:511] Master 127.2.182.62:44427 requested a full tablet report, sending...
I20260924 10:44:46.515451 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--builtin_ntp_servers=127.2.182.20:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:44:46.515980 7405 ts_manager.cc:194] Registered new tserver with Master: 01ca8d57260a4f46ab257a14b7b29f3b (127.2.182.1:38175)
W20260924 10:44:46.625653 7596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:46.625955 7596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:46.626036 7596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:46.630004 7596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:46.630074 7596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:46.630159 7596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:46.634721 7596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:46.635718 7596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:46.637066 7596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:46.646135 7602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.646109 7601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.646100 7604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.646175 7596 server_base.cc:1034] running on GCE node
I20260924 10:44:46.646819 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:46.647382 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:46.648571 7596 hybrid_clock.cc:648] HybridClock initialized: now 1790246686648556 us; error 30 us; skew 500 ppm
I20260924 10:44:46.650280 7596 webserver.cc:460] Webserver started at http://127.2.182.2:35855/ using document root <none> and password file <none>
I20260924 10:44:46.650818 7596 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:46.650919 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:46.651137 7596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:46.653092 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "6ca4a3ec860c4e848b5e0054990738db"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "3e74d7258270baf3cd695bb10ddb36d2"
server_key_iv: "a211848a9a98d2992618be53156d143c"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.653617 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "6ca4a3ec860c4e848b5e0054990738db"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "3e74d7258270baf3cd695bb10ddb36d2"
server_key_iv: "a211848a9a98d2992618be53156d143c"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.657047 7596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:44:46.659278 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.660356 7596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:46.660480 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "6ca4a3ec860c4e848b5e0054990738db"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "3e74d7258270baf3cd695bb10ddb36d2"
server_key_iv: "a211848a9a98d2992618be53156d143c"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.660583 7596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:46.664871 7596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:46.665453 7596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:46.665619 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:46.666122 7596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:46.666942 7596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:46.667009 7596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.667070 7596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:46.667119 7596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.675731 7596 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34315
I20260924 10:44:46.675773 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34315 every 8 connection(s)
I20260924 10:44:46.676574 7596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:46.681806 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7596
I20260924 10:44:46.681936 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:46.682264 2776 external_mini_cluster.cc:1428] Setting key 145efd0fa85a90d9e743719b27f11cf8
I20260924 10:44:46.682484 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.678325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:49707 (local address 127.2.182.62:44427)
0924 10:44:46.678457 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.678461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.679427 (+ 966us) server_negotiation.cc:407] Connection header received
0924 10:44:46.680707 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.680710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.680753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.680858 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.682018 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.682022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.682029 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.682043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.682062 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.682070 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.682149 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.682323 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.682372 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:46.683290 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.678690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44427 (local address 127.2.182.2:49707)
0924 10:44:46.679291 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.679327 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.680500 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.681053 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.681059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.681140 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.681653 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.681656 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.681708 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.681808 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.682158 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.682167 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.682174 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.682346 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.682563 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260924 10:44:46.684230 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--builtin_ntp_servers=127.2.182.20:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:44:46.684362 7724 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44427
I20260924 10:44:46.684587 7724 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:46.685083 7724 heartbeater.cc:511] Master 127.2.182.62:44427 requested a full tablet report, sending...
I20260924 10:44:46.686148 7405 ts_manager.cc:194] Registered new tserver with Master: 6ca4a3ec860c4e848b5e0054990738db (127.2.182.2:34315)
W20260924 10:44:46.792101 7727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:46.792414 7727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:46.792454 7727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:46.796710 7727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:46.796767 7727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:46.796845 7727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:46.801450 7727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:46.802368 7727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:46.803629 7727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:46.812551 7735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:46.812536 7732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.812767 7727 server_base.cc:1034] running on GCE node
W20260924 10:44:46.812826 7733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:46.813290 7727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:46.813875 7727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:46.815063 7727 hybrid_clock.cc:648] HybridClock initialized: now 1790246686815054 us; error 34 us; skew 500 ppm
I20260924 10:44:46.816756 7727 webserver.cc:460] Webserver started at http://127.2.182.3:33379/ using document root <none> and password file <none>
I20260924 10:44:46.817363 7727 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:46.817454 7727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:46.817678 7727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:46.819619 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "442d0dac3570433cad737b5c4a5b7344"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "dd64065637a0b7e2a766bfd7ebb655c0"
server_key_iv: "aab04702b23b953ae7cd9a2a5654df8b"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.820211 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "442d0dac3570433cad737b5c4a5b7344"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "dd64065637a0b7e2a766bfd7ebb655c0"
server_key_iv: "aab04702b23b953ae7cd9a2a5654df8b"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.823740 7727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:46.826052 7741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.827178 7727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:46.827315 7727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "442d0dac3570433cad737b5c4a5b7344"
format_stamp: "Formatted at 2026-09-24 10:44:46 on dist-test-slave-6lwt"
server_key: "dd64065637a0b7e2a766bfd7ebb655c0"
server_key_iv: "aab04702b23b953ae7cd9a2a5654df8b"
server_key_version: "encryptionkey@0"
I20260924 10:44:46.827418 7727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:46.831910 7727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:46.832453 7727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:46.832623 7727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:46.833175 7727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:46.833983 7727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:46.834049 7727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.834111 7727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:46.834146 7727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:46.842857 7727 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38501
I20260924 10:44:46.842885 7854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:38501 every 8 connection(s)
I20260924 10:44:46.843667 7727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:46.849321 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.845357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:51107 (local address 127.2.182.62:44427)
0924 10:44:46.845483 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.845487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.846461 (+ 974us) server_negotiation.cc:407] Connection header received
0924 10:44:46.847723 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.847726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.847770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.847870 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.848835 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.848839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.848842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.848873 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.848926 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.848935 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.849015 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.849147 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.849186 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:44:46.850042 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.845730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44427 (local address 127.2.182.3:51107)
0924 10:44:46.846329 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.846365 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.847526 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.848016 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.848022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.848099 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.848590 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.848591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.848626 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.848685 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.849002 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.849010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.849015 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.849183 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.849397 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260924 10:44:46.850417 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7727
I20260924 10:44:46.850535 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:46.850790 2776 external_mini_cluster.cc:1428] Setting key f74e2c7c1d8a9dc88d4c95fdc19c7fea
I20260924 10:44:46.851203 7855 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44427
I20260924 10:44:46.851416 7855 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:46.851830 7855 heartbeater.cc:511] Master 127.2.182.62:44427 requested a full tablet report, sending...
I20260924 10:44:46.852826 7405 ts_manager.cc:194] Registered new tserver with Master: 442d0dac3570433cad737b5c4a5b7344 (127.2.182.3:38501)
I20260924 10:44:46.864295 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:44:46.867542 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.866130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50502 (local address 127.2.182.62:44427)
0924 10:44:46.866239 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.866242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.866447 (+ 205us) server_negotiation.cc:407] Connection header received
0924 10:44:46.866538 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.866540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.866566 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.866629 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.867226 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.867230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.867232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.867242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.867257 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.867264 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.867329 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.867425 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.867456 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:44:46.870164 7405 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:46.873051 7405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:46.883842 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.880820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:38175 (local address 127.0.0.1:59432)
0924 10:44:46.881829 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.881842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.881909 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.882381 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.882386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.882464 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.883046 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.883051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.883075 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.883146 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.883483 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.883494 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.883500 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.883619 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.883682 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":103,"threads_started":1}
I20260924 10:44:46.883875 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.881484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.2.182.1:38175)
0924 10:44:46.881855 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.881870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.881921 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:44:46.882006 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.882013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.882204 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.882716 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.883310 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.883318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.883330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.883344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.883382 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.883417 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.883531 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.883631 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.883709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":152,"threads_started":1}
I20260924 10:44:46.884246 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.880721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34315 (local address 127.0.0.1:42230)
0924 10:44:46.881489 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.881533 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.881632 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.883172 (+ 1540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.883175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.883190 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.883593 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.883597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.883609 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.883633 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.884007 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.884010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.884013 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.884077 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.884111 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":75,"threads_started":1}
I20260924 10:44:46.884507 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.881087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:38501 (local address 127.0.0.1:58692)
0924 10:44:46.881927 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.881941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.882018 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.882678 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.882684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.882703 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.883294 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.883296 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.883305 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.883327 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.884284 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.884287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.884290 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.884364 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.884395 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":89,"threads_started":1}
I20260924 10:44:46.884709 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.880842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.2.182.2:34315)
0924 10:44:46.882141 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.882150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.882171 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:44:46.882216 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.882221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.882384 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.882518 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.883766 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.883772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.883785 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.883797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.883834 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.883862 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.883966 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.884495 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.884559 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":127,"threads_started":1}
I20260924 10:44:46.885301 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.881315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.2.182.3:38501)
0924 10:44:46.881682 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.881700 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.882087 (+ 387us) server_negotiation.cc:407] Connection header received
0924 10:44:46.882364 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.882370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.882548 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.883195 (+ 647us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.884043 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.884049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.884058 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.884070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.884107 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.884133 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.884246 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.884654 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.884723 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260924 10:44:46.886952 7527 tablet_service.cc:1467] Processing CreateTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 (DEFAULT_TABLE table=test-table [id=26c5660fc67f41cb81c549915941cc04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:46.887286 7658 tablet_service.cc:1467] Processing CreateTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 (DEFAULT_TABLE table=test-table [id=26c5660fc67f41cb81c549915941cc04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:46.887933 7527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a053c33c0be24ea996c4c0db252bbfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:46.887759 7789 tablet_service.cc:1467] Processing CreateTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 (DEFAULT_TABLE table=test-table [id=26c5660fc67f41cb81c549915941cc04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:46.888221 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a053c33c0be24ea996c4c0db252bbfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:46.888639 7789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a053c33c0be24ea996c4c0db252bbfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:46.893926 7872 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Bootstrap starting.
I20260924 10:44:46.895514 7873 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Bootstrap starting.
I20260924 10:44:46.895617 7874 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Bootstrap starting.
I20260924 10:44:46.896413 7872 tablet_bootstrap.cc:654] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:46.897289 7872 log.cc:824] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:46.897683 7873 tablet_bootstrap.cc:654] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:46.898643 7873 log.cc:824] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:46.899382 7872 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: No bootstrap required, opened a new log
I20260924 10:44:46.899382 7874 tablet_bootstrap.cc:654] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:46.899634 7872 ts_tablet_manager.cc:1397] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:46.900149 7874 log.cc:824] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:46.901026 7873 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: No bootstrap required, opened a new log
I20260924 10:44:46.901194 7873 ts_tablet_manager.cc:1397] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:46.902060 7874 tablet_bootstrap.cc:492] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: No bootstrap required, opened a new log
I20260924 10:44:46.902231 7874 ts_tablet_manager.cc:1397] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 10:44:46.903749 7872 raft_consensus.cc:348] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.903985 7872 raft_consensus.cc:374] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:46.904059 7872 raft_consensus.cc:729] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ca8d57260a4f46ab257a14b7b29f3b, State: Initialized, Role: FOLLOWER
I20260924 10:44:46.904498 7872 consensus_queue.cc:260] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.905139 7593 heartbeater.cc:503] Master 127.2.182.62:44427 was elected leader, sending a full tablet report...
I20260924 10:44:46.905508 7872 ts_tablet_manager.cc:1428] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:46.905637 7873 raft_consensus.cc:348] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.905869 7873 raft_consensus.cc:374] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:46.905911 7873 raft_consensus.cc:729] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca4a3ec860c4e848b5e0054990738db, State: Initialized, Role: FOLLOWER
I20260924 10:44:46.906350 7873 consensus_queue.cc:260] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.906529 7874 raft_consensus.cc:348] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.906764 7874 raft_consensus.cc:374] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:46.906858 7874 raft_consensus.cc:729] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442d0dac3570433cad737b5c4a5b7344, State: Initialized, Role: FOLLOWER
I20260924 10:44:46.907238 7874 consensus_queue.cc:260] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.907349 7873 ts_tablet_manager.cc:1428] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:44:46.907630 7724 heartbeater.cc:503] Master 127.2.182.62:44427 was elected leader, sending a full tablet report...
I20260924 10:44:46.908077 7874 ts_tablet_manager.cc:1428] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 10:44:46.908351 7855 heartbeater.cc:503] Master 127.2.182.62:44427 was elected leader, sending a full tablet report...
W20260924 10:44:46.927604 7725 tablet.cc:2369] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:46.966776 7879 raft_consensus.cc:482] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:46.967001 7879 raft_consensus.cc:504] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.968128 7879 leader_election.cc:288] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01ca8d57260a4f46ab257a14b7b29f3b (127.2.182.1:38175), 442d0dac3570433cad737b5c4a5b7344 (127.2.182.3:38501)
I20260924 10:44:46.970091 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.968336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:38175 (local address 127.2.182.2:47377)
0924 10:44:46.968497 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.968513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.968591 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.968966 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.968970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.968993 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.969491 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.969493 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.969507 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.969539 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.969811 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.969831 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.969838 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.969923 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.969959 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:44:46.970141 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.968565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:47377 (local address 127.2.182.1:38175)
0924 10:44:46.968693 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.968697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.968712 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:46.968760 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.968764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.968802 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.968912 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.969670 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.969673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.969676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.969688 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.969711 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.969721 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.969817 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.969943 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.969976 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:44:46.970317 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.968349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:38501 (local address 127.2.182.2:43599)
0924 10:44:46.968941 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:46.968956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:46.969066 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:46.969320 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:46.969323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:46.969336 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:46.969724 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:44:46.969726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:46.969735 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:44:46.969764 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:46.970032 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:46.970036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:46.970039 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:44:46.970116 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:44:46.970147 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":185,"thread_start_us":140,"threads_started":1}
I20260924 10:44:46.970327 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:46.968463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:43599 (local address 127.2.182.3:38501)
0924 10:44:46.968594 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:46.968600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:46.969039 (+ 439us) server_negotiation.cc:407] Connection header received
0924 10:44:46.969171 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:46.969175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:46.969209 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:46.969302 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:46.969896 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:46.969903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:46.969907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:46.969919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:46.969942 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:46.969952 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:46.970033 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:46.970115 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:44:46.970173 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:46.970861 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a053c33c0be24ea996c4c0db252bbfc2" candidate_uuid: "6ca4a3ec860c4e848b5e0054990738db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" is_pre_election: true
I20260924 10:44:46.970991 7809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a053c33c0be24ea996c4c0db252bbfc2" candidate_uuid: "6ca4a3ec860c4e848b5e0054990738db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442d0dac3570433cad737b5c4a5b7344" is_pre_election: true
I20260924 10:44:46.971172 7547 raft_consensus.cc:2454] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca4a3ec860c4e848b5e0054990738db in term 0.
I20260924 10:44:46.971210 7809 raft_consensus.cc:2454] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca4a3ec860c4e848b5e0054990738db in term 0.
I20260924 10:44:46.971632 7612 leader_election.cc:302] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ca8d57260a4f46ab257a14b7b29f3b, 6ca4a3ec860c4e848b5e0054990738db; no voters:
I20260924 10:44:46.971840 7879 raft_consensus.cc:2781] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:46.971926 7879 raft_consensus.cc:482] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:46.971957 7879 raft_consensus.cc:3037] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.973491 7879 raft_consensus.cc:504] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.973839 7879 leader_election.cc:288] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [CANDIDATE]: Term 1 election: Requested vote from peers 01ca8d57260a4f46ab257a14b7b29f3b (127.2.182.1:38175), 442d0dac3570433cad737b5c4a5b7344 (127.2.182.3:38501)
I20260924 10:44:46.974145 7809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a053c33c0be24ea996c4c0db252bbfc2" candidate_uuid: "6ca4a3ec860c4e848b5e0054990738db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442d0dac3570433cad737b5c4a5b7344"
I20260924 10:44:46.974145 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a053c33c0be24ea996c4c0db252bbfc2" candidate_uuid: "6ca4a3ec860c4e848b5e0054990738db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ca8d57260a4f46ab257a14b7b29f3b"
I20260924 10:44:46.974249 7809 raft_consensus.cc:3037] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.974298 7547 raft_consensus.cc:3037] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:46.975243 7809 raft_consensus.cc:2454] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca4a3ec860c4e848b5e0054990738db in term 1.
I20260924 10:44:46.975389 7547 raft_consensus.cc:2454] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca4a3ec860c4e848b5e0054990738db in term 1.
I20260924 10:44:46.975546 7614 leader_election.cc:302] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442d0dac3570433cad737b5c4a5b7344, 6ca4a3ec860c4e848b5e0054990738db; no voters:
I20260924 10:44:46.975713 7879 raft_consensus.cc:2781] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:46.975926 7879 raft_consensus.cc:686] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 1 LEADER]: Becoming Leader. State: Replica: 6ca4a3ec860c4e848b5e0054990738db, State: Running, Role: LEADER
I20260924 10:44:46.976298 7879 consensus_queue.cc:237] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } }
I20260924 10:44:46.979249 7404 catalog_manager.cc:5495] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ca4a3ec860c4e848b5e0054990738db (127.2.182.2). New cstate: current_term: 1 leader_uuid: "6ca4a3ec860c4e848b5e0054990738db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca4a3ec860c4e848b5e0054990738db" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 } health_report { overall_health: UNKNOWN } } }
W20260924 10:44:47.006830 7594 tablet.cc:2369] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:47.009215 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.007526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.2.182.2:34315)
0924 10:44:47.007788 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.007792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.007807 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:47.007978 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.007981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.008041 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.008133 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:47.008748 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:47.008751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:47.008754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:47.008765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:47.008791 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:47.008799 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:47.008889 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:47.009032 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 10:44:47.009072 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260924 10:44:47.014744 7547 raft_consensus.cc:1261] T a053c33c0be24ea996c4c0db252bbfc2 P 01ca8d57260a4f46ab257a14b7b29f3b [term 1 FOLLOWER]: Refusing update from remote peer 6ca4a3ec860c4e848b5e0054990738db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:47.014770 7809 raft_consensus.cc:1261] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 1 FOLLOWER]: Refusing update from remote peer 6ca4a3ec860c4e848b5e0054990738db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:47.015408 7879 consensus_queue.cc:1035] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ca8d57260a4f46ab257a14b7b29f3b" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 38175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:47.015626 7882 consensus_queue.cc:1035] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [LEADER]: Connected to new peer: Peer: permanent_uuid: "442d0dac3570433cad737b5c4a5b7344" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:47.022099 7885 mvcc.cc:205] Tried to move back new op lower bound from 7332850430005972992 to 7332850429857615872. Current Snapshot: MvccSnapshot[applied={T|T < 7332850430005972992}]
I20260924 10:44:47.023252 7886 mvcc.cc:205] Tried to move back new op lower bound from 7332850430005972992 to 7332850429857615872. Current Snapshot: MvccSnapshot[applied={T|T < 7332850430005972992}]
I20260924 10:44:47.024082 7887 mvcc.cc:205] Tried to move back new op lower bound from 7332850430005972992 to 7332850429857615872. Current Snapshot: MvccSnapshot[applied={T|T < 7332850430005972992}]
I20260924 10:44:47.031044 7405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:47.031220 7405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:47.033383 7405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b4d7b97f05b43c7bc70bd805c98823d: Sending DeleteTablet for 3 replicas of tablet a053c33c0be24ea996c4c0db252bbfc2
I20260924 10:44:47.034077 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:47 UTC) from {username='slave'} at 127.0.0.1:42230
I20260924 10:44:47.034070 7527 tablet_service.cc:1514] Processing DeleteTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:47 UTC) from {username='slave'} at 127.0.0.1:59432
I20260924 10:44:47.034513 7895 tablet_replica.cc:330] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: stopping tablet replica
I20260924 10:44:47.034690 7789 tablet_service.cc:1514] Processing DeleteTablet for tablet a053c33c0be24ea996c4c0db252bbfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:47 UTC) from {username='slave'} at 127.0.0.1:58692
I20260924 10:44:47.034837 7895 raft_consensus.cc:2229] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:47.035179 7895 raft_consensus.cc:2258] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:47.035260 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7465
I20260924 10:44:47.035338 7896 tablet_replica.cc:330] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: stopping tablet replica
I20260924 10:44:47.035610 7896 raft_consensus.cc:2229] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:47.035871 7896 raft_consensus.cc:2258] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:47.037328 7896 ts_tablet_manager.cc:1905] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:47.037388 7895 ts_tablet_manager.cc:1905] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:47.039984 7896 ts_tablet_manager.cc:1918] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:47.040107 7896 log.cc:1196] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/a053c33c0be24ea996c4c0db252bbfc2
I20260924 10:44:47.040416 7895 ts_tablet_manager.cc:1918] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:47.040443 7896 ts_tablet_manager.cc:1939] T a053c33c0be24ea996c4c0db252bbfc2 P 442d0dac3570433cad737b5c4a5b7344: Deleting consensus metadata
I20260924 10:44:47.040535 7895 log.cc:1196] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/a053c33c0be24ea996c4c0db252bbfc2
I20260924 10:44:47.040882 7895 ts_tablet_manager.cc:1939] T a053c33c0be24ea996c4c0db252bbfc2 P 6ca4a3ec860c4e848b5e0054990738db: Deleting consensus metadata
I20260924 10:44:47.041579 7392 catalog_manager.cc:4841] TS 442d0dac3570433cad737b5c4a5b7344 (127.2.182.3:38501): tablet a053c33c0be24ea996c4c0db252bbfc2 (table test-table [id=26c5660fc67f41cb81c549915941cc04]) successfully deleted
I20260924 10:44:47.041966 7389 catalog_manager.cc:4841] TS 6ca4a3ec860c4e848b5e0054990738db (127.2.182.2:34315): tablet a053c33c0be24ea996c4c0db252bbfc2 (table test-table [id=26c5660fc67f41cb81c549915941cc04]) successfully deleted
W20260924 10:44:47.044824 7390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:38175 (error 108)
I20260924 10:44:47.045636 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7596
I20260924 10:44:47.045881 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.045552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:38175 (local address 127.0.0.1:59444)
0924 10:44:47.045695 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:47.045784 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:44:47.049199 7390 catalog_manager.cc:4584] TS 01ca8d57260a4f46ab257a14b7b29f3b (127.2.182.1:38175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a053c33c0be24ea996c4c0db252bbfc2: Network error: Client connection negotiation failed: client connection to 127.2.182.1:38175: connect: Connection refused (error 111)
I20260924 10:44:47.051003 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7727
I20260924 10:44:47.056418 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7374
2026-09-24T10:44:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (967 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 10:44:47.077832 2776 mini_oidc.cc:152] Starting JWKS server
I20260924 10:44:47.079334 2776 webserver.cc:460] Webserver started at https://127.0.0.1:42423/ using document root <none> and password file <none>
I20260924 10:44:47.079504 2776 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:44:47.079777 2776 webserver.cc:460] Webserver started at http://127.0.0.1:44865/ using document root <none> and password file <none>
2026-09-24T10:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:47Z Disabled control of system clock
I20260924 10:44:47.088442 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42059
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:42059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42423/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/testchainca.cert with env {}
W20260924 10:44:47.197499 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:47.197739 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:47.197777 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:47.201612 7903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:47.201663 7903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:47.201676 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:47.201711 7903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:47.201731 7903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:44:47.201747 7903 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:44:47.201761 7903 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42423/jwks.json
I20260924 10:44:47.206918 7903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:42059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42423/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:47.207821 7903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:47.208719 7903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:47.216925 7911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:47.216908 7908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:47.216934 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:47.217175 7903 server_base.cc:1034] running on GCE node
I20260924 10:44:47.217818 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:47.218868 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:47.220045 7903 hybrid_clock.cc:648] HybridClock initialized: now 1790246687220032 us; error 24 us; skew 500 ppm
I20260924 10:44:47.221774 7903 webserver.cc:460] Webserver started at http://127.2.182.62:43353/ using document root <none> and password file <none>
I20260924 10:44:47.222354 7903 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:47.222443 7903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:47.222656 7903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:47.224627 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "cfcc494adc574c9bab0eca2b7d96e02e"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "ca4850a0425a697f2509ed7a2dda9242"
server_key_iv: "91b04837d8bec76ee0ad4a39b21e3582"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.225191 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "cfcc494adc574c9bab0eca2b7d96e02e"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "ca4850a0425a697f2509ed7a2dda9242"
server_key_iv: "91b04837d8bec76ee0ad4a39b21e3582"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.228575 7903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:47.230839 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:47.231951 7903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:47.232084 7903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "cfcc494adc574c9bab0eca2b7d96e02e"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "ca4850a0425a697f2509ed7a2dda9242"
server_key_iv: "91b04837d8bec76ee0ad4a39b21e3582"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.232185 7903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:47.253384 7903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:47.254032 7903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:47.254216 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:47.260792 7903 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:42059
I20260924 10:44:47.260797 7969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:42059 every 8 connection(s)
I20260924 10:44:47.261664 7903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:47.263969 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 7903
I20260924 10:44:47.264071 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:47.264364 2776 external_mini_cluster.cc:1428] Setting key e0627a8a687043550f23c75007f0b868
I20260924 10:44:47.264784 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:47.271330 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: Bootstrap starting.
I20260924 10:44:47.273164 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.265637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.2.182.62:42059)
0924 10:44:47.266226 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.266244 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.266279 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:44:47.266901 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.266917 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.267289 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.267551 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:47.267881 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.268800 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.270016 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.270484 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.270606 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.271438 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:47.271445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:47.271462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:47.271839 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:47.271891 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:47.271910 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:47.271995 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:47.272152 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:44:47.272414 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260924 10:44:47.273762 7970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:47.274616 7970 log.cc:824] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:47.276023 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:47.276607 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: No bootstrap required, opened a new log
I20260924 10:44:47.280050 7970 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } }
I20260924 10:44:47.280308 7970 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:47.280371 7970 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcc494adc574c9bab0eca2b7d96e02e, State: Initialized, Role: FOLLOWER
I20260924 10:44:47.280807 7970 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } }
I20260924 10:44:47.280995 7970 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:47.281049 7970 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:47.281118 7970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:47.281857 7970 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } }
I20260924 10:44:47.282127 7970 leader_election.cc:302] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfcc494adc574c9bab0eca2b7d96e02e; no voters:
I20260924 10:44:47.282351 7970 leader_election.cc:288] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:47.282438 7975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:47.282684 7975 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [term 1 LEADER]: Becoming Leader. State: Replica: cfcc494adc574c9bab0eca2b7d96e02e, State: Running, Role: LEADER
I20260924 10:44:47.283004 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } }
I20260924 10:44:47.283520 7970 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:47.284682 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfcc494adc574c9bab0eca2b7d96e02e. Latest consensus state: current_term: 1 leader_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } } }
I20260924 10:44:47.284715 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc494adc574c9bab0eca2b7d96e02e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42059 } } }
I20260924 10:44:47.284817 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:47.284817 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:47.288491 7991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:47.288568 7991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:47.288628 7984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:47.289376 7984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:47.293956 7984 catalog_manager.cc:1347] Generated new cluster ID: 904145bff6fb414f9e57d43136e596da
I20260924 10:44:47.294019 7984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:47.315806 7984 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:47.317046 7984 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:47.331071 7984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfcc494adc574c9bab0eca2b7d96e02e: Generated new TSK 0
I20260924 10:44:47.331887 7984 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:44:47.369290 8000 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 10:44:47.369417 8000 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:44:47.369851 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.345815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.2.182.62:42059)
0924 10:44:47.345980 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.345984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.346114 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:44:47.346333 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.346335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.346377 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.346458 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:44:47.346736 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.347240 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.348487 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.348694 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.348770 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.349161 (+ 391us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:44:47.369485 (+ 20324us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:44:47.369739 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 10:44:47.369930 7973 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 10:44:47.371285 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:47.345664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:42059 (local address 127.0.0.1:55142)
0924 10:44:47.346022 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:47.346037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:47.346166 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:47.346497 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:47.346500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:47.346509 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:44:47.346633 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.346639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.347361 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.347365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.348352 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.348362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.348802 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.348806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.348930 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.348980 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:44:47.369731 (+ 20751us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:44:47.369974 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.62:42059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.182.62:42059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 10:44:47.372239 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 7903
2026-09-24T10:44:47Z chronyd exiting
I20260924 10:44:47.408738 2776 test_util.cc:175] -----------------------------------------------
I20260924 10:44:47.408839 2776 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246662595599-2776-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 10:44:47.409880 2776 mini_oidc.cc:152] Starting JWKS server
I20260924 10:44:47.410657 2776 webserver.cc:460] Webserver started at https://127.0.0.1:43831/ using document root <none> and password file <none>
I20260924 10:44:47.410738 2776 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:44:47.410933 2776 webserver.cc:460] Webserver started at http://127.0.0.1:41981/ using document root <none> and password file <none>
2026-09-24T10:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:47Z Disabled control of system clock
I20260924 10:44:47.419267 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36285
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43831/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/testchainca.cert with env {}
W20260924 10:44:47.533710 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:47.533958 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:47.534027 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:47.538013 8006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:47.538067 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:47.538082 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:47.538133 8006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:47.538159 8006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:44:47.538193 8006 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:44:47.538233 8006 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43831/jwks.json
I20260924 10:44:47.543428 8006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43831/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:47.544368 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:47.545365 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:47.553349 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:47.553334 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:47.553450 8006 server_base.cc:1034] running on GCE node
W20260924 10:44:47.553351 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:47.554121 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:47.555154 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:47.556339 8006 hybrid_clock.cc:648] HybridClock initialized: now 1790246687556324 us; error 24 us; skew 500 ppm
I20260924 10:44:47.558055 8006 webserver.cc:460] Webserver started at http://127.2.182.62:45417/ using document root <none> and password file <none>
I20260924 10:44:47.558610 8006 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:47.558699 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:47.558921 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:47.560849 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "41d9acbe180042ff90a643a27bc963c2"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "63003f59d7d4a040261e9951e1856b89"
server_key_iv: "6c5c27cf451d606ab3d1fa2b396b193a"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.561455 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "41d9acbe180042ff90a643a27bc963c2"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "63003f59d7d4a040261e9951e1856b89"
server_key_iv: "6c5c27cf451d606ab3d1fa2b396b193a"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.565016 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:47.567277 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:47.568732 8006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:47.568851 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "41d9acbe180042ff90a643a27bc963c2"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "63003f59d7d4a040261e9951e1856b89"
server_key_iv: "6c5c27cf451d606ab3d1fa2b396b193a"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.568992 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:47.579555 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:47.580128 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:47.580309 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:47.587013 8006 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36285
I20260924 10:44:47.587016 8072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36285 every 8 connection(s)
I20260924 10:44:47.587903 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:47.590833 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:47.594445 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8006
I20260924 10:44:47.594570 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:47.594837 2776 external_mini_cluster.cc:1428] Setting key 492a1573fdfe8a6a0c34b37bcbaf41a3
I20260924 10:44:47.596419 8073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2: Bootstrap starting.
I20260924 10:44:47.598793 8073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:47.599510 8073 log.cc:824] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:47.601671 8073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2: No bootstrap required, opened a new log
I20260924 10:44:47.603885 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.596197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.2.182.62:36285)
0924 10:44:47.596782 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.596797 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.596831 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:47.597540 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.597556 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.597942 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.598214 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:47.598560 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.599584 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.600735 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.601250 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.601372 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.602238 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:47.602243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:47.602254 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:47.602609 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:47.602660 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:47.602679 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:47.602750 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:47.602921 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:44:47.603228 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":129,"threads_started":1}
I20260924 10:44:47.605525 8073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } }
I20260924 10:44:47.605808 8073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:47.605901 8073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d9acbe180042ff90a643a27bc963c2, State: Initialized, Role: FOLLOWER
I20260924 10:44:47.606385 8073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } }
I20260924 10:44:47.606534 8073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:47.606600 8073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:47.606702 8073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:47.606838 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:47.607791 8073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } }
I20260924 10:44:47.608129 8073 leader_election.cc:302] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d9acbe180042ff90a643a27bc963c2; no voters:
I20260924 10:44:47.608434 8073 leader_election.cc:288] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:47.608531 8078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:47.608757 8078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [term 1 LEADER]: Becoming Leader. State: Replica: 41d9acbe180042ff90a643a27bc963c2, State: Running, Role: LEADER
I20260924 10:44:47.609089 8078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } }
I20260924 10:44:47.609606 8073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:47.610661 8080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d9acbe180042ff90a643a27bc963c2. Latest consensus state: current_term: 1 leader_uuid: "41d9acbe180042ff90a643a27bc963c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } } }
I20260924 10:44:47.610839 8080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:47.611178 8079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d9acbe180042ff90a643a27bc963c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9acbe180042ff90a643a27bc963c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36285 } } }
I20260924 10:44:47.611248 8087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:47.611980 8079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:47.614794 8087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:47.619513 8087 catalog_manager.cc:1347] Generated new cluster ID: ca165e8f109e40d4972beb116150a285
I20260924 10:44:47.619567 8087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:47.638482 8087 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:47.639678 8087 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:47.648782 8087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d9acbe180042ff90a643a27bc963c2: Generated new TSK 0
I20260924 10:44:47.649562 8087 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:44:47.674446 8103 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 10:44:47.674532 8103 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:44:47.675133 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.659603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.2.182.62:36285)
0924 10:44:47.659726 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.659732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.659867 (+ 135us) server_negotiation.cc:407] Connection header received
0924 10:44:47.660022 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.660025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.660070 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.660155 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:44:47.660408 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.660990 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.662182 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.662396 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.662469 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.662768 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:44:47.674826 (+ 12058us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:44:47.675028 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53028: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:44:47.675215 8076 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53028: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 10:44:47.675374 8102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:47.659450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36285 (local address 127.0.0.1:53028)
0924 10:44:47.659772 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:47.659786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:47.659897 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:47.660175 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:47.660178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:47.660186 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:44:47.660298 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.660305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.661091 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.661095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.662068 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.662080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.662499 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.662503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.662592 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.662620 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:44:47.675006 (+ 12386us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:44:47.675199 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.62:36285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20260924 10:44:47.675920 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8006
2026-09-24T10:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 10:44:47.715118 2776 mini_oidc.cc:152] Starting JWKS server
I20260924 10:44:47.715922 2776 webserver.cc:460] Webserver started at https://127.0.0.1:33477/ using document root <none> and password file <none>
I20260924 10:44:47.716003 2776 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:44:47.716254 2776 webserver.cc:460] Webserver started at http://127.0.0.1:44687/ using document root <none> and password file <none>
2026-09-24T10:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:47Z Disabled control of system clock
I20260924 10:44:47.724583 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35581
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:46779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:35581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33477/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:44:47.828850 8109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:47.829118 8109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:47.829154 8109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:47.833038 8109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:47.833086 8109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:47.833101 8109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:47.833135 8109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:47.833150 8109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:44:47.833165 8109 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:44:47.833178 8109 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33477/jwks.json
I20260924 10:44:47.837644 8109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:35581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33477/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:47.838532 8109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:47.839818 8109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:47.847929 8109 server_base.cc:1034] running on GCE node
W20260924 10:44:47.847882 8114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:47.847894 8117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:47.847877 8115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:47.848690 8109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:47.849782 8109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:47.850965 8109 hybrid_clock.cc:648] HybridClock initialized: now 1790246687850940 us; error 41 us; skew 500 ppm
I20260924 10:44:47.852748 8109 webserver.cc:460] Webserver started at http://127.2.182.62:35267/ using document root <none> and password file <none>
I20260924 10:44:47.853353 8109 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:47.853453 8109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:47.853736 8109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:47.855679 8109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "94e67a2f4bb04c17b898302182dc18a1"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "9bab3183ca65affac9463512529f2090"
server_key_iv: "4a2898ad5e921517c9adb05527ccbeb2"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.856217 8109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "94e67a2f4bb04c17b898302182dc18a1"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "9bab3183ca65affac9463512529f2090"
server_key_iv: "4a2898ad5e921517c9adb05527ccbeb2"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.859826 8109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:47.862115 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:47.863276 8109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:47.863430 8109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "94e67a2f4bb04c17b898302182dc18a1"
format_stamp: "Formatted at 2026-09-24 10:44:47 on dist-test-slave-6lwt"
server_key: "9bab3183ca65affac9463512529f2090"
server_key_iv: "4a2898ad5e921517c9adb05527ccbeb2"
server_key_version: "encryptionkey@0"
I20260924 10:44:47.863538 8109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:47.886942 8109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:47.887575 8109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:47.887759 8109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:47.894522 8109 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:35581
I20260924 10:44:47.894531 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:35581 every 8 connection(s)
I20260924 10:44:47.895426 8109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:47.898456 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:47.899650 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8109
I20260924 10:44:47.899772 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:47.900159 2776 external_mini_cluster.cc:1428] Setting key b1811ba9e04f85d0e36c1f3878b50aba
I20260924 10:44:47.905652 8176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1: Bootstrap starting.
I20260924 10:44:47.908169 8176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:47.909045 8176 log.cc:824] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:47.909837 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.901636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.2.182.62:35581)
0924 10:44:47.902452 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.902472 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.902514 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:44:47.903451 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.903470 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.903837 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.904132 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:47.904457 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.905471 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.906633 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.907076 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.907189 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.907987 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:47.907992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:47.908004 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:47.908406 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:47.908460 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:47.908480 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:47.908553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:47.908710 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:44:47.909018 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":128,"threads_started":1}
I20260924 10:44:47.910800 8176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1: No bootstrap required, opened a new log
I20260924 10:44:47.912411 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:47.914497 8176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } }
I20260924 10:44:47.914687 8176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:47.914719 8176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e67a2f4bb04c17b898302182dc18a1, State: Initialized, Role: FOLLOWER
I20260924 10:44:47.915052 8176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } }
I20260924 10:44:47.915161 8176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:47.915205 8176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:47.915269 8176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:47.916014 8176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } }
I20260924 10:44:47.916280 8176 leader_election.cc:302] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94e67a2f4bb04c17b898302182dc18a1; no voters:
I20260924 10:44:47.916579 8176 leader_election.cc:288] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:47.916692 8181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:47.916940 8181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [term 1 LEADER]: Becoming Leader. State: Replica: 94e67a2f4bb04c17b898302182dc18a1, State: Running, Role: LEADER
I20260924 10:44:47.917301 8181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } }
I20260924 10:44:47.917847 8176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:47.918944 8182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94e67a2f4bb04c17b898302182dc18a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } } }
I20260924 10:44:47.919081 8182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:47.919394 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e67a2f4bb04c17b898302182dc18a1. Latest consensus state: current_term: 1 leader_uuid: "94e67a2f4bb04c17b898302182dc18a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e67a2f4bb04c17b898302182dc18a1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35581 } } }
I20260924 10:44:47.919459 8190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:47.919548 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:47.921942 8190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:47.926945 8190 catalog_manager.cc:1347] Generated new cluster ID: 0aed2f6dd2914a9190655bfe2431e4a9
I20260924 10:44:47.927047 8190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:47.937099 8190 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:47.938236 8190 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:47.943535 8190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94e67a2f4bb04c17b898302182dc18a1: Generated new TSK 0
I20260924 10:44:47.944124 8190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:44:47.956516 8205 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:47.951521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:35581 (local address 127.0.0.1:53314)
0924 10:44:47.951820 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:47.951831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:47.951927 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:47.952202 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:47.952204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:47.952210 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:44:47.952294 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.952298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.953311 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.953314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.953891 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:47.953897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.954451 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.954453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:47.954512 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.954531 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:44:47.956209 (+ 1678us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:44:47.956323 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.62:35581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260924 10:44:47.957185 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:47.951628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.2.182.62:35581)
0924 10:44:47.951750 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:47.951755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:47.951898 (+ 143us) server_negotiation.cc:407] Connection header received
0924 10:44:47.952024 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:47.952027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:47.952080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:47.952201 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:44:47.952386 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.953197 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.954004 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:47.954336 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:47.954430 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:47.954675 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:44:47.956001 (+ 1326us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:44:47.957058 (+ 1057us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 10:44:47.957403 8179 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260924 10:44:47.957587 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8109
E20260924 10:44:47.967751 8206 webserver.cc:533] Webserver: sslize failed:
E20260924 10:44:47.967824 8206 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T10:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T10:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:47Z Disabled control of system clock
I20260924 10:44:48.003917 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37813
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:37813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/testchainca.cert with env {}
W20260924 10:44:48.109902 8210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:48.110163 8210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:48.110230 8210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:48.113978 8210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:48.114049 8210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:48.114078 8210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:48.114130 8210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:48.114172 8210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:44:48.114205 8210 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:44:48.114238 8210 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 10:44:48.119335 8210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:37813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:48.120337 8210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:48.121416 8210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:48.129585 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:48.129644 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:48.129585 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:48.129853 8210 server_base.cc:1034] running on GCE node
I20260924 10:44:48.130512 8210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:48.131578 8210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:48.132762 8210 hybrid_clock.cc:648] HybridClock initialized: now 1790246688132738 us; error 46 us; skew 500 ppm
I20260924 10:44:48.134531 8210 webserver.cc:460] Webserver started at http://127.2.182.62:44331/ using document root <none> and password file <none>
I20260924 10:44:48.135103 8210 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:48.135202 8210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:48.135427 8210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:48.137463 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "37842c7107a649e489c5e2c886ff28e2"
format_stamp: "Formatted at 2026-09-24 10:44:48 on dist-test-slave-6lwt"
server_key: "f858920f485f2025f47b57741485c8e2"
server_key_iv: "9228a4cf5865f29f7328006e4890dc83"
server_key_version: "encryptionkey@0"
I20260924 10:44:48.138032 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "37842c7107a649e489c5e2c886ff28e2"
format_stamp: "Formatted at 2026-09-24 10:44:48 on dist-test-slave-6lwt"
server_key: "f858920f485f2025f47b57741485c8e2"
server_key_iv: "9228a4cf5865f29f7328006e4890dc83"
server_key_version: "encryptionkey@0"
I20260924 10:44:48.141505 8210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:48.143756 8224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:48.145031 8210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:48.145159 8210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "37842c7107a649e489c5e2c886ff28e2"
format_stamp: "Formatted at 2026-09-24 10:44:48 on dist-test-slave-6lwt"
server_key: "f858920f485f2025f47b57741485c8e2"
server_key_iv: "9228a4cf5865f29f7328006e4890dc83"
server_key_version: "encryptionkey@0"
I20260924 10:44:48.145265 8210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:48.157742 8210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:48.158288 8210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:48.158461 8210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:48.165091 8210 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:37813
I20260924 10:44:48.165130 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:37813 every 8 connection(s)
I20260924 10:44:48.165937 8210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:48.168740 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:48.169623 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8210
I20260924 10:44:48.169761 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:48.170049 2776 external_mini_cluster.cc:1428] Setting key d272b82562750a0fde517d5e3eafe2c8
I20260924 10:44:48.174317 8277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: Bootstrap starting.
I20260924 10:44:48.176921 8277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:48.177673 8277 log.cc:824] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:48.178857 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:48.171335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.2.182.62:37813)
0924 10:44:48.171893 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:48.171907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:48.171940 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:44:48.172509 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:48.172524 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:48.172855 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:48.173195 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:48.173558 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.174459 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.175652 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.176091 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.176200 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:48.177077 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:48.177082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:48.177094 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:48.177457 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:48.177507 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:48.177525 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:48.177598 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:48.177779 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 10:44:48.178071 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260924 10:44:48.179595 8277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: No bootstrap required, opened a new log
I20260924 10:44:48.181352 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:48.183282 8277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } }
I20260924 10:44:48.183508 8277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:48.183585 8277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37842c7107a649e489c5e2c886ff28e2, State: Initialized, Role: FOLLOWER
I20260924 10:44:48.184012 8277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } }
I20260924 10:44:48.184155 8277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:48.184243 8277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:48.184337 8277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:48.185297 8277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } }
I20260924 10:44:48.185627 8277 leader_election.cc:302] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37842c7107a649e489c5e2c886ff28e2; no voters:
I20260924 10:44:48.185909 8277 leader_election.cc:288] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:48.186017 8282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:48.186239 8282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [term 1 LEADER]: Becoming Leader. State: Replica: 37842c7107a649e489c5e2c886ff28e2, State: Running, Role: LEADER
I20260924 10:44:48.186590 8282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } }
I20260924 10:44:48.187067 8277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:48.188246 8284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37842c7107a649e489c5e2c886ff28e2. Latest consensus state: current_term: 1 leader_uuid: "37842c7107a649e489c5e2c886ff28e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } } }
I20260924 10:44:48.188374 8284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:48.188627 8283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37842c7107a649e489c5e2c886ff28e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37842c7107a649e489c5e2c886ff28e2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 37813 } } }
I20260924 10:44:48.188706 8283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:48.191395 8297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:48.191463 8297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:48.191538 8291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:48.192181 8291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:48.197093 8291 catalog_manager.cc:1347] Generated new cluster ID: 686bf7c43aec4e968461947999dce42e
I20260924 10:44:48.197150 8291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:48.223460 8291 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:48.224714 8291 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:48.233096 8291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37842c7107a649e489c5e2c886ff28e2: Generated new TSK 0
I20260924 10:44:48.233872 8291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:48.345036 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:48.243316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.2.182.62:37813)
0924 10:44:48.243463 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:48.243469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:48.243885 (+ 416us) server_negotiation.cc:407] Connection header received
0924 10:44:48.244065 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:48.244069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:48.244126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:48.244242 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:44:48.244552 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.245414 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.246672 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.247065 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.247137 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:48.247560 (+ 423us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:44:48.344555 (+ 96995us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:44:48.344809 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53896: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 10:44:48.345108 8280 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53896: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 10:44:48.345237 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:48.243452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:37813 (local address 127.0.0.1:53896)
0924 10:44:48.243772 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:48.243791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:48.243927 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:48.244253 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:48.244256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:48.244267 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:44:48.244427 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:48.244437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.245550 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.245554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:48.246536 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:48.246545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:48.247257 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:48.247260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:48.247351 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:48.247375 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:44:48.344763 (+ 97388us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:44:48.345026 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.62:37813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260924 10:44:48.345817 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8210
2026-09-24T10:44:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (369 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:48 dist-test-slave-6lwt krb5kdc[8318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:48 dist-test-slave-6lwt krb5kdc[8318](info): set up 2 sockets
Sep 24 10:44:48 dist-test-slave-6lwt krb5kdc[8318](info): commencing operation
krb5kdc: starting...
W20260924 10:44:49.609383 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:49 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:44:49.725931 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43745
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:43745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:49.831554 8331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:49.831799 8331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:49.831836 8331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:49.835645 8331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:49.835695 8331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:49.835713 8331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:49.835729 8331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:49.835744 8331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:49.840595 8331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:43745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:49.841601 8331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:49.842564 8331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:49.850662 8339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:49.850709 8331 server_base.cc:1034] running on GCE node
W20260924 10:44:49.850677 8336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:49.850720 8337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:49.851428 8331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:49.852456 8331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:49.853647 8331 hybrid_clock.cc:648] HybridClock initialized: now 1790246689853629 us; error 49 us; skew 500 ppm
Sep 24 10:44:49 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:49.856184 8331 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:44:49.857070 8331 webserver.cc:460] Webserver started at http://127.2.182.62:33923/ using document root <none> and password file <none>
I20260924 10:44:49.857577 8331 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:49.857658 8331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:49.857870 8331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:49.859731 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "664c17bf4a364796b207f960bdd408c2"
format_stamp: "Formatted at 2026-09-24 10:44:49 on dist-test-slave-6lwt"
server_key: "d7a5abbe62d37a51bdc46e7e2ba9068e"
server_key_iv: "6532aee5508d39aadd0d729ace6fbc9f"
server_key_version: "encryptionkey@0"
I20260924 10:44:49.860248 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "664c17bf4a364796b207f960bdd408c2"
format_stamp: "Formatted at 2026-09-24 10:44:49 on dist-test-slave-6lwt"
server_key: "d7a5abbe62d37a51bdc46e7e2ba9068e"
server_key_iv: "6532aee5508d39aadd0d729ace6fbc9f"
server_key_version: "encryptionkey@0"
I20260924 10:44:49.863678 8331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:49.865931 8346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:49.867007 8331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:49.867127 8331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "664c17bf4a364796b207f960bdd408c2"
format_stamp: "Formatted at 2026-09-24 10:44:49 on dist-test-slave-6lwt"
server_key: "d7a5abbe62d37a51bdc46e7e2ba9068e"
server_key_iv: "6532aee5508d39aadd0d729ace6fbc9f"
server_key_version: "encryptionkey@0"
I20260924 10:44:49.867220 8331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:49.877897 8331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:49.880395 8331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:49.880576 8331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:49.887302 8331 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:43745
I20260924 10:44:49.887321 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:43745 every 8 connection(s)
I20260924 10:44:49.888118 8331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:49.890630 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:49.891667 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8331
I20260924 10:44:49.891798 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:49.892105 2776 external_mini_cluster.cc:1428] Setting key fd8f819448f9507b97ee445401832ca4
I20260924 10:44:49.896163 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2: Bootstrap starting.
I20260924 10:44:49.898272 8399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:49.899109 8399 log.cc:824] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:44:49 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:49.901218 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2: No bootstrap required, opened a new log
I20260924 10:44:49.905305 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:49.893381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.2.182.62:43745)
0924 10:44:49.893767 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:49.893781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:49.893813 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:44:49.894388 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:49.894402 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:49.894684 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:49.895002 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:49.895781 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:49.896743 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:49.897601 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:49.898046 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:49.898208 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:49.901018 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:49.901023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:49.901034 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:49.901057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:49.902573 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:49.903095 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:49.903099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:49.903103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:49.903201 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:49.903522 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:49.903524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:49.903525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:49.903853 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:49.903996 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:49.904272 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:44:49.904566 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20260924 10:44:49.905184 8399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } }
I20260924 10:44:49.905418 8399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:49.905479 8399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664c17bf4a364796b207f960bdd408c2, State: Initialized, Role: FOLLOWER
I20260924 10:44:49.905890 8399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } }
I20260924 10:44:49.906047 8399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:49.906122 8399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:49.906244 8399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:49.907032 8399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } }
I20260924 10:44:49.907367 8399 leader_election.cc:302] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664c17bf4a364796b207f960bdd408c2; no voters:
I20260924 10:44:49.907667 8399 leader_election.cc:288] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:49.907776 8404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:49.907965 8404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [term 1 LEADER]: Becoming Leader. State: Replica: 664c17bf4a364796b207f960bdd408c2, State: Running, Role: LEADER
I20260924 10:44:49.908233 8404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } }
I20260924 10:44:49.908897 8399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:49.909385 8406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664c17bf4a364796b207f960bdd408c2. Latest consensus state: current_term: 1 leader_uuid: "664c17bf4a364796b207f960bdd408c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } } }
I20260924 10:44:49.909919 8406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:49.910696 8413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:49.910909 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "664c17bf4a364796b207f960bdd408c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664c17bf4a364796b207f960bdd408c2" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43745 } } }
I20260924 10:44:49.911059 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:49.913647 8413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:49.918810 8413 catalog_manager.cc:1347] Generated new cluster ID: 624183bade474b648525d58effff359f
I20260924 10:44:49.918867 8413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:49.932358 8413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:49.933271 8413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:49.943316 8413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 664c17bf4a364796b207f960bdd408c2: Generated new TSK 0
I20260924 10:44:49.943966 8413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:44:49.971308 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:50.077536 8425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:50.077754 8425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:50.077795 8425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:50.081583 8425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:50.081629 8425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:50.081724 8425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:50.086880 8425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:50.087772 8425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:50.088676 8425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:50.097743 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:50.097919 8433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:50.097985 8425 server_base.cc:1034] running on GCE node
W20260924 10:44:50.098109 8430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:50.098503 8425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:50.099109 8425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:50.100296 8425 hybrid_clock.cc:648] HybridClock initialized: now 1790246690100257 us; error 44 us; skew 500 ppm
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.102859 8425 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:44:50.103691 8425 webserver.cc:460] Webserver started at http://127.2.182.1:37977/ using document root <none> and password file <none>
I20260924 10:44:50.104207 8425 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:50.104308 8425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:50.104525 8425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:50.106562 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "df3dae1608194d3babf4e0a6decfd1a2"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "17f7e6e2a5c29944ce4ed665119f2d2f"
server_key_iv: "8f3534c8782eb11336c7816268e51ec1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.107106 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "df3dae1608194d3babf4e0a6decfd1a2"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "17f7e6e2a5c29944ce4ed665119f2d2f"
server_key_iv: "8f3534c8782eb11336c7816268e51ec1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.110795 8425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:44:50.113173 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.114302 8425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:44:50.114429 8425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "df3dae1608194d3babf4e0a6decfd1a2"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "17f7e6e2a5c29944ce4ed665119f2d2f"
server_key_iv: "8f3534c8782eb11336c7816268e51ec1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.114532 8425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:50.136471 8425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:50.138976 8425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:50.139156 8425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:50.139667 8425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:50.140409 8425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:50.140486 8425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.140548 8425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:50.140588 8425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.149573 8425 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:37731
I20260924 10:44:50.149600 8553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:37731 every 8 connection(s)
I20260924 10:44:50.150406 8425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:50.157225 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8425
I20260924 10:44:50.157384 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:50.157733 2776 external_mini_cluster.cc:1428] Setting key 3dddccc88fe8b36ee464fc4f3bb50705
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:44:50.170405 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.152214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:43833 (local address 127.2.182.62:43745)
0924 10:44:50.152400 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.152405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.153346 (+ 941us) server_negotiation.cc:407] Connection header received
0924 10:44:50.154614 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.154617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.154665 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.154745 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.155834 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.156488 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.157741 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.158086 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.158200 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.166607 (+ 8407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.166612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.166618 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.166653 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.168136 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.168665 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.168667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.168669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.168747 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.169121 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.169126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.169128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.169425 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.169525 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.170178 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.170288 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:44:50.171229 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.152657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43745 (local address 127.2.182.1:43833)
0924 10:44:50.153202 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:50.153242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:50.154421 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:50.154889 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:50.154896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:50.155250 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:50.155698 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.155710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.156594 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.156596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.157622 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.157631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.158244 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.158250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.158418 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.164995 (+ 6577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:50.165013 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:50.166417 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:50.168239 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.168243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.168255 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.168554 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.168823 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.168825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.168826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.169037 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.169543 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:50.169549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:50.169891 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 10:44:50.170087 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 10:44:50.170443 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":187,"threads_started":1}
I20260924 10:44:50.172290 8554 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43745
I20260924 10:44:50.172521 8554 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:50.173086 8554 heartbeater.cc:511] Master 127.2.182.62:43745 requested a full tablet report, sending...
I20260924 10:44:50.174671 8363 ts_manager.cc:194] Registered new tserver with Master: df3dae1608194d3babf4e0a6decfd1a2 (127.2.182.1:37731)
I20260924 10:44:50.176298 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:43833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:44:50.183861 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:50.284824 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:50.285120 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:50.285189 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:50.288959 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:50.289002 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:50.289089 8559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:50.293711 8559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:50.294559 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:50.295466 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:50.304157 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:50.304167 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:50.304204 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:50.304356 8559 server_base.cc:1034] running on GCE node
I20260924 10:44:50.304739 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:50.305364 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:50.306515 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790246690306505 us; error 22 us; skew 500 ppm
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.309036 8559 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:44:50.309953 8559 webserver.cc:460] Webserver started at http://127.2.182.2:44873/ using document root <none> and password file <none>
I20260924 10:44:50.310474 8559 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:50.310559 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:50.310765 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:50.312613 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "c2dbdd3f08ef463a87922278dd823cf4"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "a8714578a596a0e8083a2a401081ec26"
server_key_iv: "e356f6bda42f47ef4719a8130c6f7be1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.313151 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "c2dbdd3f08ef463a87922278dd823cf4"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "a8714578a596a0e8083a2a401081ec26"
server_key_iv: "e356f6bda42f47ef4719a8130c6f7be1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.316439 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:44:50.318859 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.319931 8559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:50.320040 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "c2dbdd3f08ef463a87922278dd823cf4"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "a8714578a596a0e8083a2a401081ec26"
server_key_iv: "e356f6bda42f47ef4719a8130c6f7be1"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.320132 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:50.334739 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:50.337301 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:50.337472 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:50.337980 8559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:50.338699 8559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:50.338742 8559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.338817 8559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:50.338855 8559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.347543 8559 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:35415
I20260924 10:44:50.347574 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:35415 every 8 connection(s)
I20260924 10:44:50.348351 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:50.349576 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8559
I20260924 10:44:50.349681 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:50.350000 2776 external_mini_cluster.cc:1428] Setting key 825b6f528fbc8ac22210006a3aabc60c
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
I20260924 10:44:50.361860 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.350053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:42539 (local address 127.2.182.62:43745)
0924 10:44:50.350185 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.350189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.351043 (+ 854us) server_negotiation.cc:407] Connection header received
0924 10:44:50.352449 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.352452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.352504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.352582 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.353779 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.354290 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.355415 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.355659 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.355772 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.358167 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.358170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.358172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.358193 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.359628 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.360219 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.360223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.360225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.360303 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.360674 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.360678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.360680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.360855 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.361030 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.361580 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.361695 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:50.362597 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.350472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43745 (local address 127.2.182.2:42539)
0924 10:44:50.350907 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:50.350953 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:50.352246 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:50.352755 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:50.352761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:50.353200 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:50.353637 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.353650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.354414 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.354417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.355287 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.355296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.355803 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.355807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.355982 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.356581 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:50.356596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:50.358006 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:50.359781 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.359794 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.359807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.360132 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.360396 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.360400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.360402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.360577 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.361038 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:50.361046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:50.361346 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 10:44:50.361530 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:44:50.361822 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
I20260924 10:44:50.363821 8688 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43745
I20260924 10:44:50.364051 8688 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:50.364535 8688 heartbeater.cc:511] Master 127.2.182.62:43745 requested a full tablet report, sending...
I20260924 10:44:50.365660 8363 ts_manager.cc:194] Registered new tserver with Master: c2dbdd3f08ef463a87922278dd823cf4 (127.2.182.2:35415)
I20260924 10:44:50.366492 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:42539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:44:50.375622 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:50.481369 8693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:50.481586 8693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:50.481653 8693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:50.485724 8693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:50.485771 8693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:50.485865 8693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:50.490679 8693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:50.491544 8693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:50.492401 8693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:50.501156 8701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:50.501150 8699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:50.501286 8693 server_base.cc:1034] running on GCE node
W20260924 10:44:50.501196 8698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:50.501752 8693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:50.502327 8693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:50.503535 8693 hybrid_clock.cc:648] HybridClock initialized: now 1790246690503504 us; error 41 us; skew 500 ppm
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.506150 8693 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:44:50.506887 8693 webserver.cc:460] Webserver started at http://127.2.182.3:40991/ using document root <none> and password file <none>
I20260924 10:44:50.507368 8693 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:50.507454 8693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:50.507655 8693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:50.509649 8693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "504e3b9cd7af4d76bbc67e1f18f9ae3c"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "e61a5cdc69c6bbd59035347c2b6b4c96"
server_key_iv: "af077b75ca9c6a936e0e8f3b50285970"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.510214 8693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "504e3b9cd7af4d76bbc67e1f18f9ae3c"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "e61a5cdc69c6bbd59035347c2b6b4c96"
server_key_iv: "af077b75ca9c6a936e0e8f3b50285970"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.513742 8693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:50.515914 8708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.517071 8693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:50.517176 8693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "504e3b9cd7af4d76bbc67e1f18f9ae3c"
format_stamp: "Formatted at 2026-09-24 10:44:50 on dist-test-slave-6lwt"
server_key: "e61a5cdc69c6bbd59035347c2b6b4c96"
server_key_iv: "af077b75ca9c6a936e0e8f3b50285970"
server_key_version: "encryptionkey@0"
I20260924 10:44:50.517273 8693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:50.536072 8693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:50.538579 8693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:50.538761 8693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:50.539268 8693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:50.539997 8693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:50.540041 8693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.540107 8693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:50.540144 8693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:50.548276 8693 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44995
I20260924 10:44:50.548298 8821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44995 every 8 connection(s)
I20260924 10:44:50.549127 8693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:50.550923 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8693
I20260924 10:44:50.551033 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:50.551287 2776 external_mini_cluster.cc:1428] Setting key cc3076f643ec91ffba1f1e56014166bc
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:44:50.560842 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.550728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:46777 (local address 127.2.182.62:43745)
0924 10:44:50.550872 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.550878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.551702 (+ 824us) server_negotiation.cc:407] Connection header received
0924 10:44:50.553015 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.553018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.553075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.553160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.554128 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.554623 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.555657 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.555858 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.555932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.558103 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.558106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.558108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.558126 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.559261 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.559663 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.559666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.559667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.559732 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.560015 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.560017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.560018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.560165 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.560239 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.560646 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.560729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:50.561600 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.551096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43745 (local address 127.2.182.3:46777)
0924 10:44:50.551550 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:50.551596 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:50.552809 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:50.553279 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:50.553299 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:50.553604 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:50.554013 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.554024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.554721 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.554723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.555559 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:50.555567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.555971 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.555974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:50.556098 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.556662 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:50.556677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:50.557973 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:50.559356 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.559360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.559369 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.559585 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.559809 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:50.559811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:50.559812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:50.559939 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:50.560247 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:50.560251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:50.560462 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 10:44:50.560601 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 10:44:50.560810 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20260924 10:44:50.562649 8822 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43745
I20260924 10:44:50.562847 8822 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:50.563297 8822 heartbeater.cc:511] Master 127.2.182.62:43745 requested a full tablet report, sending...
I20260924 10:44:50.564210 8363 ts_manager.cc:194] Registered new tserver with Master: 504e3b9cd7af4d76bbc67e1f18f9ae3c (127.2.182.3:44995)
I20260924 10:44:50.564848 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:46777
I20260924 10:44:50.564855 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/test-admin.keytab.
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.581199 2776 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 10:44:50.583630 8829 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:50.582331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.2.182.1:37731)
0924 10:44:50.582644 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.582653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.582672 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:44:50.582715 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.582719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.582860 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.582985 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.583434 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":140,"threads_started":1}
W20260924 10:44:50.583725 2776 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.584761 2776 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.585705 2776 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:50.594127 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.586433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.2.182.1:37731)
0924 10:44:50.586545 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.586551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.586648 (+ 97us) server_negotiation.cc:407] Connection header received
0924 10:44:50.586725 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.586726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.586758 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.586819 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.587248 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.587944 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.588696 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.589075 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.589155 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.591145 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.591152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.591162 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.591182 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.592675 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.593084 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.593089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.593090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.593168 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.593352 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.593354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.593355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.593529 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.593673 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.593815 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.593971 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 10:44:50.597337 8829 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:50.596520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.2.182.1:37731)
0924 10:44:50.596639 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.596642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.596656 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:44:50.596756 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.596758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.596796 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.596855 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.597244 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50450: BlockingRecv error: recv got EOF from 127.0.0.1:50450 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:44:50.597384 2776 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.598325 2776 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.599293 2776 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:50.607232 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.600045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.2.182.1:37731)
0924 10:44:50.600172 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.600177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.600313 (+ 136us) server_negotiation.cc:407] Connection header received
0924 10:44:50.600425 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.600428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.600468 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.600551 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.601059 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.601550 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.602334 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.602556 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.602663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.604671 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.604674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.604675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.604694 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.605978 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.606394 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.606396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.606397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.606460 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.606670 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.606672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.606673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.606811 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.606899 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.607074 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.607144 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:44:50.609504 8829 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:50.608591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.2.182.1:37731)
0924 10:44:50.608692 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.608696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.608819 (+ 123us) server_negotiation.cc:407] Connection header received
0924 10:44:50.608938 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.608943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.608979 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.609077 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.609406 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50466: BlockingRecv error: recv got EOF from 127.0.0.1:50466 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 10:44:50.609659 2776 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246662595599-2776-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.610667 2776 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:50.611466 2776 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:44:50.619398 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:50.612149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.2.182.1:37731)
0924 10:44:50.612249 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:50.612254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:50.612387 (+ 133us) server_negotiation.cc:407] Connection header received
0924 10:44:50.612458 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:50.612459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:50.612488 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:50.612545 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:50.613015 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.613447 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.614131 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:50.614314 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:50.614382 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:50.616496 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:50.616499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:50.616501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:50.616519 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:50.617892 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.618276 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.618279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.618280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.618366 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:50.618638 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:50.618640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:50.618641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:50.618836 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:50.618927 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:50.619127 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 10:44:50.619231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:44:50.620338 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8425
I20260924 10:44:50.625674 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8559
I20260924 10:44:50.631080 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8693
I20260924 10:44:50.637585 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8331
2026-09-24T10:44:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2058 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8842](info): set up 2 sockets
Sep 24 10:44:50 dist-test-slave-6lwt krb5kdc[8842](info): commencing operation
krb5kdc: starting...
W20260924 10:44:51.649446 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:51 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab.
Entry for principal oryx/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab.
I20260924 10:44:51.761961 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32999
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab
--principal=oryx/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:51.867137 8855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:51.867393 8855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:51.867431 8855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:51.871313 8855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:51.871363 8855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:51.871382 8855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:51.871416 8855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:51.871431 8855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:51.876392 8855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab
--principal=oryx/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:51.877370 8855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:51.878281 8855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:51.886660 8855 server_base.cc:1034] running on GCE node
W20260924 10:44:51.886606 8861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:51.886574 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:51.886587 8860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:51.887413 8855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:51.888455 8855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:51.889613 8855 hybrid_clock.cc:648] HybridClock initialized: now 1790246691889596 us; error 28 us; skew 500 ppm
Sep 24 10:44:51 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:51.892329 8855 init.cc:377] Logged in from keytab as oryx/127.2.182.62@KRBTEST.COM (short username oryx)
I20260924 10:44:51.893210 8855 webserver.cc:460] Webserver started at http://127.2.182.62:41991/ using document root <none> and password file <none>
I20260924 10:44:51.893747 8855 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:51.893803 8855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:51.894037 8855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:51.895951 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "4c41f027916c4d61af007de8145c955c"
format_stamp: "Formatted at 2026-09-24 10:44:51 on dist-test-slave-6lwt"
server_key: "19bebb2eaf82e0726ae74dd16c7c71bd"
server_key_iv: "2928f7ee5ac072d358a99acdc5a80e3b"
server_key_version: "encryptionkey@0"
I20260924 10:44:51.896531 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "4c41f027916c4d61af007de8145c955c"
format_stamp: "Formatted at 2026-09-24 10:44:51 on dist-test-slave-6lwt"
server_key: "19bebb2eaf82e0726ae74dd16c7c71bd"
server_key_iv: "2928f7ee5ac072d358a99acdc5a80e3b"
server_key_version: "encryptionkey@0"
I20260924 10:44:51.900012 8855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:51.902333 8870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:51.903512 8855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:51.903649 8855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4c41f027916c4d61af007de8145c955c"
format_stamp: "Formatted at 2026-09-24 10:44:51 on dist-test-slave-6lwt"
server_key: "19bebb2eaf82e0726ae74dd16c7c71bd"
server_key_iv: "2928f7ee5ac072d358a99acdc5a80e3b"
server_key_version: "encryptionkey@0"
I20260924 10:44:51.903757 8855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:51.920428 8855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:51.923223 8855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:51.923420 8855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:51.930346 8855 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32999
I20260924 10:44:51.930610 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32999 every 8 connection(s)
I20260924 10:44:51.931324 8855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:51.934089 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:51.937146 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8855
I20260924 10:44:51.937276 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:51.937563 2776 external_mini_cluster.cc:1428] Setting key 3394910485a8ca5840cd67fb46565b97
I20260924 10:44:51.939944 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Bootstrap starting.
I20260924 10:44:51.942561 8923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:51.943306 8923 log.cc:824] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:51.945191 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: No bootstrap required, opened a new log
Sep 24 10:44:51 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:51.948931 8923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } }
I20260924 10:44:51.949146 8923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:51.949218 8923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c41f027916c4d61af007de8145c955c, State: Initialized, Role: FOLLOWER
I20260924 10:44:51.949694 8923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } }
I20260924 10:44:51.949860 8923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:51.949930 8923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:51.950039 8923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:51.951125 8923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } }
I20260924 10:44:51.951458 8923 leader_election.cc:302] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c41f027916c4d61af007de8145c955c; no voters:
I20260924 10:44:51.951748 8923 leader_election.cc:288] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:51.951848 8928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:51.952065 8928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 LEADER]: Becoming Leader. State: Replica: 4c41f027916c4d61af007de8145c955c, State: Running, Role: LEADER
I20260924 10:44:51.952207 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:51.938964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.2.182.62:32999)
0924 10:44:51.939370 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:51.939386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:51.939422 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:44:51.940024 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:51.940039 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:51.940409 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:51.940688 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:51.941633 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:51.942598 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:51.943485 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:51.944135 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:51.944299 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:51.947251 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:51.947256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:51.947268 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:51.947292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:51.949093 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:51.949590 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:51.949595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:51.949599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:51.949705 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:51.950030 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:51.950033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:51.950035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:51.950479 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:51.950639 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:51.950926 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:44:51.951193 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260924 10:44:51.952380 8928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } }
I20260924 10:44:51.953056 8923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:51.953984 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } }
I20260924 10:44:51.954101 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:51.954398 8930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c41f027916c4d61af007de8145c955c. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } }
I20260924 10:44:51.954523 8930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:51.954792 8937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:51.957779 8937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:51.963023 8937 catalog_manager.cc:1347] Generated new cluster ID: c9e4592b017045d2a59c0acecdbdfe92
I20260924 10:44:51.963084 8937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:51.974082 8937 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:51.975193 8937 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:51.980641 8937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Generated new TSK 0
I20260924 10:44:51.981448 8937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
I20260924 10:44:52.012198 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:52.120944 8949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:52.121172 8949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:52.121254 8949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:52.125200 8949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:52.125247 8949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:52.125355 8949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:52.130345 8949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:52.131286 8949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:52.132243 8949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:52.141266 8957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.141500 8955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.141500 8954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:52.141588 8949 server_base.cc:1034] running on GCE node
I20260924 10:44:52.141953 8949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:52.142568 8949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:52.143746 8949 hybrid_clock.cc:648] HybridClock initialized: now 1790246692143733 us; error 25 us; skew 500 ppm
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:52.146338 8949 init.cc:377] Logged in from keytab as oryx/127.2.182.1@KRBTEST.COM (short username oryx)
I20260924 10:44:52.147120 8949 webserver.cc:460] Webserver started at http://127.2.182.1:44031/ using document root <none> and password file <none>
I20260924 10:44:52.147667 8949 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:52.147753 8949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:52.147981 8949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:52.149974 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "e3ace47fb8494fe7addfe54733d314dc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "9c42db551faa4415ba344b48410cc86a"
server_key_iv: "161d495376824d195085911c8d1a0678"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.150516 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "e3ace47fb8494fe7addfe54733d314dc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "9c42db551faa4415ba344b48410cc86a"
server_key_iv: "161d495376824d195085911c8d1a0678"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.155797 8949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:44:52.159078 8964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.160756 8949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:44:52.160874 8949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "e3ace47fb8494fe7addfe54733d314dc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "9c42db551faa4415ba344b48410cc86a"
server_key_iv: "161d495376824d195085911c8d1a0678"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.161017 8949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:52.183465 8949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:52.186287 8949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:52.186483 8949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:52.187084 8949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:52.187954 8949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:52.187999 8949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.188036 8949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:52.188056 8949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.198475 8949 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:43595
I20260924 10:44:52.198520 9077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:43595 every 8 connection(s)
I20260924 10:44:52.199424 8949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:52.208251 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 8949
I20260924 10:44:52.208388 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:52.208730 2776 external_mini_cluster.cc:1428] Setting key b668f17f35806e3f901e61626b26e240
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:52.214146 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.201361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:45073 (local address 127.2.182.62:32999)
0924 10:44:52.201500 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.201509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.202379 (+ 870us) server_negotiation.cc:407] Connection header received
0924 10:44:52.203740 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.203744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.203801 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.203909 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.205126 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.205677 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.206812 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.207056 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.207163 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.210162 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.210165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.210168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.210194 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.211684 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.212364 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.212370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.212372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.212464 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.212803 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.212807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.212809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.213070 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.213303 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.213926 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.214029 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:52.215405 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.201766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:45073)
0924 10:44:52.202219 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.202267 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.203524 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.204081 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.204088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.204448 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.204981 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.204996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.205803 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.205806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.206686 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.206694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.207214 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.207218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.207377 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.208097 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.208120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.209977 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.211971 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.211979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.211994 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.212257 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.212556 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.212560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.212562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.212704 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.213311 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.213319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.213656 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.213878 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.214239 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260924 10:44:52.216763 9078 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:52.217051 9078 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:52.217525 9078 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:52.219588 8887 ts_manager.cc:194] Registered new tserver with Master: e3ace47fb8494fe7addfe54733d314dc (127.2.182.1:43595)
WARNING: no policy specified for oryx/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:44:52.220820 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:45073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
I20260924 10:44:52.287978 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:52.391371 9083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:52.391595 9083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:52.391631 9083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:52.395521 9083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:52.395569 9083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:52.395661 9083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:52.400277 9083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:52.401182 9083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:52.402066 9083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:52.410741 9089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.410771 9091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.410771 9088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:52.411044 9083 server_base.cc:1034] running on GCE node
I20260924 10:44:52.411338 9083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:52.411852 9083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:52.413018 9083 hybrid_clock.cc:648] HybridClock initialized: now 1790246692412993 us; error 33 us; skew 500 ppm
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:52.415443 9083 init.cc:377] Logged in from keytab as oryx/127.2.182.2@KRBTEST.COM (short username oryx)
I20260924 10:44:52.416217 9083 webserver.cc:460] Webserver started at http://127.2.182.2:39067/ using document root <none> and password file <none>
I20260924 10:44:52.416719 9083 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:52.416806 9083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:52.417068 9083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:52.418857 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "a54411e80dac479ba742837550cdedcc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "4a5b4e596cd2cb454adfe18b090f5d72"
server_key_iv: "cfb539beec0608235b55e78147b4a541"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.419399 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "a54411e80dac479ba742837550cdedcc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "4a5b4e596cd2cb454adfe18b090f5d72"
server_key_iv: "cfb539beec0608235b55e78147b4a541"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.422788 9083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:52.424999 9098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.426179 9083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:52.426304 9083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "a54411e80dac479ba742837550cdedcc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "4a5b4e596cd2cb454adfe18b090f5d72"
server_key_iv: "cfb539beec0608235b55e78147b4a541"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.426405 9083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:52.438143 9083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:52.440547 9083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:52.440724 9083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:52.441279 9083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:52.442027 9083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:52.442093 9083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.442154 9083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:52.442205 9083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.450312 9083 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:38355
I20260924 10:44:52.450361 9211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:38355 every 8 connection(s)
I20260924 10:44:52.451046 9083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:52.453085 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9083
I20260924 10:44:52.453197 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:52.453495 2776 external_mini_cluster.cc:1428] Setting key 6071647346f8e16f60f5cba123257758
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:44:52.464427 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.452702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:47441 (local address 127.2.182.62:32999)
0924 10:44:52.452855 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.452859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.454070 (+ 1211us) server_negotiation.cc:407] Connection header received
0924 10:44:52.455494 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.455497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.455548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.455629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.456711 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.457288 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.458361 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.458579 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.458685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.460960 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.460963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.460966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.460986 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.462341 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.462866 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.462870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.462872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.462954 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.463315 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.463319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.463321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.463491 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.463599 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.464090 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.464211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.3@KRBTEST.COM" created.
I20260924 10:44:52.465263 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.453044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:47441)
0924 10:44:52.453937 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.453971 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.455302 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.455799 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.455807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.456144 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.456580 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.456592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.457406 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.457409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.458228 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.458237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.458714 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.458718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.458889 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.459457 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.459472 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.460780 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.462459 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.462464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.462478 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.462770 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.463073 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.463077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.463083 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.463220 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.463595 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.463601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.463868 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.464049 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.464383 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":123,"threads_started":1}
I20260924 10:44:52.466344 9212 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:52.466591 9212 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:52.467183 9212 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:52.468304 8887 ts_manager.cc:194] Registered new tserver with Master: a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
I20260924 10:44:52.468840 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:47441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
I20260924 10:44:52.480553 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:52.585651 9217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:52.585870 9217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:52.585908 9217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:52.589888 9217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:52.589932 9217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:52.589984 9217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:52.594805 9217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:52.595685 9217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:52.596565 9217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:52.605587 9217 server_base.cc:1034] running on GCE node
W20260924 10:44:52.605548 9222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.605509 9225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.605548 9223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:52.606186 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:52.606715 9217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:52.607867 9217 hybrid_clock.cc:648] HybridClock initialized: now 1790246692607832 us; error 46 us; skew 500 ppm
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:52.610469 9217 init.cc:377] Logged in from keytab as oryx/127.2.182.3@KRBTEST.COM (short username oryx)
I20260924 10:44:52.611181 9217 webserver.cc:460] Webserver started at http://127.2.182.3:37131/ using document root <none> and password file <none>
I20260924 10:44:52.611635 9217 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:52.611687 9217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:52.611829 9217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:52.613677 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "04be2753ef0b4fd79f23ab480a3c141a"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "c2702ecb571f6e1b6a032f299e25bc6e"
server_key_iv: "48a0dda20436bf5972ede1d8e4949b07"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.614143 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "04be2753ef0b4fd79f23ab480a3c141a"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "c2702ecb571f6e1b6a032f299e25bc6e"
server_key_iv: "48a0dda20436bf5972ede1d8e4949b07"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.617487 9217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:44:52.619534 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.620636 9217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:44:52.620747 9217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "04be2753ef0b4fd79f23ab480a3c141a"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "c2702ecb571f6e1b6a032f299e25bc6e"
server_key_iv: "48a0dda20436bf5972ede1d8e4949b07"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.620839 9217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:52.645869 9217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:52.648402 9217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:52.648605 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:52.649185 9217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:52.649926 9217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:52.649978 9217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.650043 9217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:52.650084 9217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.659602 9217 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38709
I20260924 10:44:52.659626 9345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:38709 every 8 connection(s)
I20260924 10:44:52.660439 9217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:52.666608 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9217
I20260924 10:44:52.666729 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:44:52.667028 2776 external_mini_cluster.cc:1428] Setting key e85a04e17d35443140290503b40f9644
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:52.673346 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.662228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:38019 (local address 127.2.182.62:32999)
0924 10:44:52.662391 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.662394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.663128 (+ 734us) server_negotiation.cc:407] Connection header received
0924 10:44:52.664405 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.664407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.664452 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.664524 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.665679 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.666190 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.667710 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.667974 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.668077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.670521 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.670524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.670525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.670547 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.671721 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.672125 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.672127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.672128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.672193 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.672452 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.672454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.672455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.672595 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.672662 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.673148 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.673234 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:44:52.674067 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.662548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:38019)
0924 10:44:52.662951 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.663002 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.664208 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.664670 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.664676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.665078 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.665562 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.665574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.666326 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.666332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.667569 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.667583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.668098 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.668120 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.668308 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.668856 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.668870 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.670369 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.671818 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.671822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.671831 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.672051 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.672269 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.672271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.672272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.672383 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.672669 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.672673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.672868 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.673108 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.673325 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260924 10:44:52.675146 9346 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:52.675357 9346 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:52.675740 9346 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:52.676664 8887 ts_manager.cc:194] Registered new tserver with Master: 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:52.677255 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:38019
I20260924 10:44:52.680540 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.182.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.61.keytab.
Entry for principal oryx/127.2.182.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.61.keytab.
I20260924 10:44:52.707765 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:39393
--webserver_interface=127.2.182.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.61.keytab
--principal=oryx/127.2.182.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32999,127.2.182.61:39393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:52.816358 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:52.816597 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:52.816635 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:52.820505 9351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:52.820556 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:52.820575 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:52.820590 9351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:52.820640 9351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:52.825788 9351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32999,127.2.182.61:39393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.61.keytab
--principal=oryx/127.2.182.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:39393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
W20260924 10:44:52.825970 9351 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.182.62:32999,127.2.182.61:39393'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 10:44:52.826746 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:52.827733 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:52.835603 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.835647 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:52.835613 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:52.835774 9351 server_base.cc:1034] running on GCE node
I20260924 10:44:52.836396 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:52.837503 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:52.838689 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790246692838677 us; error 28 us; skew 500 ppm
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:52.841389 9351 init.cc:377] Logged in from keytab as oryx/127.2.182.61@KRBTEST.COM (short username oryx)
I20260924 10:44:52.842163 9351 webserver.cc:460] Webserver started at http://127.2.182.61:43745/ using document root <none> and password file <none>
I20260924 10:44:52.842685 9351 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:52.842741 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:52.842943 9351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:52.844998 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/instance:
uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "1660343a5294ecbf127fd1f5dc6c4703"
server_key_iv: "6a4afd6c128e2bd0b839676ec4f79ad4"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.845541 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal/instance:
uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "1660343a5294ecbf127fd1f5dc6c4703"
server_key_iv: "6a4afd6c128e2bd0b839676ec4f79ad4"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.848974 9351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:44:52.851161 9366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.852283 9351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:44:52.852432 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "1660343a5294ecbf127fd1f5dc6c4703"
server_key_iv: "6a4afd6c128e2bd0b839676ec4f79ad4"
server_key_version: "encryptionkey@0"
I20260924 10:44:52.852532 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:52.867897 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:52.870606 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:52.870792 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:52.877142 9351 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.61:39393
I20260924 10:44:52.877168 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.61:39393 every 8 connection(s)
I20260924 10:44:52.877974 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
I20260924 10:44:52.880442 9419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:52.883406 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9351
I20260924 10:44:52.884416 9419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.61@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:52.896508 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.886230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.2.182.62:32999)
0924 10:44:52.886379 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.886384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.886644 (+ 260us) server_negotiation.cc:407] Connection header received
0924 10:44:52.886852 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.886855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.886909 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.886981 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.888244 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.888811 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.889874 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.890131 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.890203 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.892886 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.892891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.892897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.892944 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.894289 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.894881 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.894884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.894886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.894967 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.895295 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.895305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.895306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.895550 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.895625 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.896195 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.896323 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:44:52.897267 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.884873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.2.182.61:39393)
0924 10:44:52.885299 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.885320 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.885354 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:52.885950 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.885965 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.886321 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.887569 (+ 1248us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.887935 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.889156 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.889988 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.890387 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.890480 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.893451 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.893457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.893464 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.893484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.895317 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.895749 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.895753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.895754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.895827 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.896114 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.896120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.896123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.896383 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.896814 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.896935 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.897070 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":10624,"thread_start_us":124,"threads_started":1}
I20260924 10:44:52.897272 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.886163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33054)
0924 10:44:52.886536 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.886583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.886727 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.887053 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.887063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.887501 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.888108 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.888120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.888940 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.888943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.889754 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.889767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.890250 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.890253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.890436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.891183 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.891197 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.892727 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.894431 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.894444 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.894453 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.894785 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.895060 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.895062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.895063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.895182 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.895656 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.895662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.895948 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.896143 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.896379 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":37,"thread_start_us":135,"threads_started":1}
I20260924 10:44:52.898514 9419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246692, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.61@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:52.905961 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.898925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44066)
0924 10:44:52.899040 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.899055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.899125 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.899408 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.899411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.899555 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.899624 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.899628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.900608 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.900611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.901134 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.901141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.901563 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.901566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.901655 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.901983 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.901993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.903228 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.904709 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.904712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.904713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.904961 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.905211 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.905214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.905216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.905295 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.905605 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.905607 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.905679 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.905760 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.905848 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:44:52.906021 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.899091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.2.182.61:39393)
0924 10:44:52.899214 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.899218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.899229 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:44:52.899259 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.899261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.899307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.899399 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.899726 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.900499 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.901233 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.901468 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.901568 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.903386 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.903389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.903391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.903410 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.904613 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.905054 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.905057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.905058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.905128 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.905371 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.905373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.905374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.905519 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.905590 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.905815 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.905897 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:44:52.908653 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Bootstrap starting.
I20260924 10:44:52.910750 9419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:52.911499 9419 log.cc:824] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:52.913367 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: No bootstrap required, opened a new log
I20260924 10:44:52.916108 9419 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } }
I20260924 10:44:52.916338 9419 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:52.916417 9419 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5acbcf659a64f5ea7e8858d307bb03f, State: Initialized, Role: FOLLOWER
I20260924 10:44:52.916819 9419 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } }
I20260924 10:44:52.917271 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } } }
I20260924 10:44:52.917376 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:52.918083 9419 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:44:52.922958 9436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:52.923064 9436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:52.930718 9425 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:52.933606 9425 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } }
I20260924 10:44:52.934374 9425 leader_election.cc:288] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999)
I20260924 10:44:52.935039 8897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c41f027916c4d61af007de8145c955c" is_pre_election: true
I20260924 10:44:52.935778 9370 leader_election.cc:302] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f; no voters: 4c41f027916c4d61af007de8145c955c
I20260924 10:44:52.936008 9425 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 10:44:52.938870 9351 master_runner.cc:428] Detected that this master a5acbcf659a64f5ea7e8858d307bb03f is joining an existing cluster
I20260924 10:44:52.938927 9351 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.182.61:39393
I20260924 10:44:52.950121 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.941489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.2.182.61:39393)
0924 10:44:52.941582 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.941586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.941865 (+ 279us) server_negotiation.cc:407] Connection header received
0924 10:44:52.942006 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.942009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.942041 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.942080 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.942931 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.943684 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.944963 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.945278 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.945341 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.946684 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.946688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.946690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.946709 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.948415 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.948807 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.948814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.948816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.948930 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.949236 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.949240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.949242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.949459 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.949574 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.949867 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.949978 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 10:44:52.950577 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.941117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44082)
0924 10:44:52.941792 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.941806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.941922 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.942185 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.942188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.942424 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.942821 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.942829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.943819 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.943821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.944823 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.944832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.945474 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.945477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.945554 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.946091 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.946111 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.946561 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.948517 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.948523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.948525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.948711 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.949047 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.949050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.949052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.949147 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.949581 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.949584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.949673 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.950289 (+ 616us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.950415 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":73,"thread_start_us":123,"threads_started":1}
I20260924 10:44:52.951215 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.941185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.2.182.62:32999)
0924 10:44:52.941402 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.941407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.941913 (+ 506us) server_negotiation.cc:407] Connection header received
0924 10:44:52.942082 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.942085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.942129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.942177 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.942784 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.943631 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.945009 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.945347 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.945417 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.946820 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.946823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.946826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.946850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.948526 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.949027 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.949033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.949035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.949139 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.949788 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.949792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.949794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.949994 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.950060 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.950965 (+ 905us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.951076 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:44:52.951761 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.941520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33062)
0924 10:44:52.941812 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.941825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.941977 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.942324 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.942327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.942540 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.942656 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.942663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.943767 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.943770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.944832 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.944842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.945560 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.945563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.945662 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.946267 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.946282 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.946675 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.948645 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.948648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.948650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.948878 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.949608 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.949611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.949613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.949686 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.950130 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.950133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.950455 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.951470 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.951597 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":62,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.2.182.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.60@KRBTEST.COM" created.
Sep 24 10:44:52 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.62@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:52.967442 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.957032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.2.182.61:39393)
0924 10:44:52.957174 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:52.957181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:52.957198 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:44:52.957290 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:52.957296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:52.957359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:52.957444 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:52.958194 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.958998 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.960231 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.960440 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.960524 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.963860 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:52.963867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:52.963873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:52.963904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:52.965455 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.965975 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.965979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.965981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.966058 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:52.966411 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:52.966414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:52.966416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:52.966613 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:52.966707 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:52.967110 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 10:44:52.967214 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:44:52.967526 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:52.956234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44092)
0924 10:44:52.956616 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:52.956673 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:52.956818 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:52.957515 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:52.957522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:52.957946 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:52.958063 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.958070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.959123 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.959127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.960113 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:52.960123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:52.960558 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:52.960562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:52.960671 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:52.961152 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:52.961166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:52.963714 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:52.965559 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.965568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.965573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.965846 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.966162 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:52.966168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:52.966173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:52.966315 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:52.966728 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:52.966737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:52.966880 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 10:44:52.967049 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 10:44:52.967272 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260924 10:44:52.969197 8887 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.182.61:39393: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
member_type: NON_VOTER
last_known_addr {
host: "127.2.182.61"
port: 39393
}
attrs {
promote: true
}
}
dest_uuid: "4c41f027916c4d61af007de8145c955c"
cas_config_opid_index: -1
I20260924 10:44:52.969799 8887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } }
I20260924 10:44:52.971568 9447 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } }
I20260924 10:44:52.971879 9393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:52.973049 9393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 10:44:52.973687 9447 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.182.61:39393
I20260924 10:44:52.973649 9448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:52.974079 9351 master.cc:537] Master@127.2.182.61:39393 shutting down...
I20260924 10:44:52.974231 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } }
I20260924 10:44:52.974337 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:52.977139 8871 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 10:44:52.978114 8871 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c -> Peer a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61:39393): Couldn't send request to peer a5acbcf659a64f5ea7e8858d307bb03f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:44:52.979089 9351 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:52.979387 9351 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:52.979497 9351 tablet_replica.cc:330] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.60.keytab.
Entry for principal oryx/127.2.182.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.60.keytab.
I20260924 10:44:52.982052 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:45581
--webserver_interface=127.2.182.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.60.keytab
--principal=oryx/127.2.182.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:52.996223 9351 master.cc:552] Master@127.2.182.61:39393 shutdown complete.
I20260924 10:44:52.996296 9351 master_runner.cc:305] Clearing existing system tablet
I20260924 10:44:52.996562 9351 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:52.996630 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:52.997748 9351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 10:44:52.998592 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:52.998827 9351 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:52.998970 9351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:52.999006 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "1660343a5294ecbf127fd1f5dc6c4703"
server_key_iv: "6a4afd6c128e2bd0b839676ec4f79ad4"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.000628 9351 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:53.002653 9351 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:44:53.002750 9351 log.cc:1196] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 10:44:53.002985 9351 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Deleting consensus metadata
I20260924 10:44:53.003180 9351 master_runner.cc:315] Copying system tablet from 127.2.182.62:32999
I20260924 10:44:53.004293 9351 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.62:32999
I20260924 10:44:53.012296 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.004831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33076)
0924 10:44:53.005197 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.005209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.005343 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.005613 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.005616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.005821 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.005919 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.005925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.006717 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.006722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.007420 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.007429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.007924 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.007928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.008021 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.008661 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.008673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.009088 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.010821 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.010824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.010826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.011018 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.011293 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.011297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.011299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.011379 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.011740 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.011743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.011849 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.011972 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.012112 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":114,"threads_started":1}
I20260924 10:44:53.012287 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.004938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.2.182.62:32999)
0924 10:44:53.005079 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.005084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.005296 (+ 212us) server_negotiation.cc:407] Connection header received
0924 10:44:53.005453 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.005456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.005511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.005597 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.006035 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.006626 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.007564 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.007813 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.007936 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.009228 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.009232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.009234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.009257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.010690 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.011113 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.011118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.011120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.011198 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.011471 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.011474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.011476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.011661 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.011752 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.012024 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.012127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:44:53.013123 8907 tablet_copy_service.cc:140] P 4c41f027916c4d61af007de8145c955c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a5acbcf659a64f5ea7e8858d307bb03f ({username='oryx', principal='oryx/127.2.182.61@KRBTEST.COM'} at 127.0.0.1:33076)
I20260924 10:44:53.013266 8907 tablet_copy_service.cc:161] P 4c41f027916c4d61af007de8145c955c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a5acbcf659a64f5ea7e8858d307bb03f at {username='oryx', principal='oryx/127.2.182.61@KRBTEST.COM'} at 127.0.0.1:33076: session id = a5acbcf659a64f5ea7e8858d307bb03f-00000000000000000000000000000000
I20260924 10:44:53.014549 8907 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:44:53.015880 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:44:53.017817 9351 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: tablet copy: Starting download of 0 data blocks...
I20260924 10:44:53.018005 9351 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: tablet copy: Starting download of 1 WAL segments...
I20260924 10:44:53.019632 9351 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:44:53.021154 8907 tablet_copy_service.cc:342] P 4c41f027916c4d61af007de8145c955c: Request end of tablet copy session a5acbcf659a64f5ea7e8858d307bb03f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.182.61@KRBTEST.COM'} at 127.0.0.1:33076
I20260924 10:44:53.021258 8907 tablet_copy_service.cc:433] P 4c41f027916c4d61af007de8145c955c: ending tablet copy session a5acbcf659a64f5ea7e8858d307bb03f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a5acbcf659a64f5ea7e8858d307bb03f
I20260924 10:44:53.027768 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:53.029348 9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.029422 9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.029649 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.030064 9351 server_base.cc:1034] running on GCE node
I20260924 10:44:53.030268 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.030611 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.031785 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790246693031772 us; error 28 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.033360 9351 init.cc:377] Logged in from keytab as oryx/127.2.182.61@KRBTEST.COM (short username oryx)
I20260924 10:44:53.033823 9351 webserver.cc:460] Webserver started at http://127.2.182.61:36683/ using document root <none> and password file <none>
I20260924 10:44:53.034000 9351 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.034055 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.035125 9351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:53.036044 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.036262 9351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:53.036331 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "1660343a5294ecbf127fd1f5dc6c4703"
server_key_iv: "6a4afd6c128e2bd0b839676ec4f79ad4"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.036414 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.064152 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.066320 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.071348 9351 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.61:39393
I20260924 10:44:53.071369 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.61:39393 every 8 connection(s)
I20260924 10:44:53.071918 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
I20260924 10:44:53.073433 9525 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:44:53.074227 9525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Bootstrap starting.
I20260924 10:44:53.081843 9525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:44:53.082137 9525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: Bootstrap complete.
I20260924 10:44:53.082613 9525 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } }
I20260924 10:44:53.082755 9525 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5acbcf659a64f5ea7e8858d307bb03f, State: Initialized, Role: LEARNER
I20260924 10:44:53.082867 9525 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } }
I20260924 10:44:53.083204 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } }
I20260924 10:44:53.083365 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: LEARNER
I20260924 10:44:53.083379 9525 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:53.086463 9539 catalog_manager.cc:1259] Loaded cluster ID: c9e4592b017045d2a59c0acecdbdfe92
I20260924 10:44:53.086556 9539 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: loading cluster ID for follower catalog manager: success
I20260924 10:44:53.088254 9539 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: acquiring CA information for follower catalog manager: success
I20260924 10:44:53.089597 9539 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:44:53.136435 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:53.136786 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:53.136857 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:53.142174 9453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:53.142241 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:53.142266 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:53.142284 9453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:53.142299 9453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:53.147684 9453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.60.keytab
--principal=oryx/127.2.182.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:45581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:53.148715 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:53.149799 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:53.160857 9544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.161033 9547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.160857 9545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.161500 9453 server_base.cc:1034] running on GCE node
I20260924 10:44:53.162098 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.163277 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.164472 9453 hybrid_clock.cc:648] HybridClock initialized: now 1790246693164438 us; error 46 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.167357 9453 init.cc:377] Logged in from keytab as oryx/127.2.182.60@KRBTEST.COM (short username oryx)
I20260924 10:44:53.168319 9453 webserver.cc:460] Webserver started at http://127.2.182.60:33541/ using document root <none> and password file <none>
I20260924 10:44:53.168871 9453 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.169011 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.169253 9453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:53.171330 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/instance:
uuid: "b6aecda326bd41458ca1e6eae80da49b"
format_stamp: "Formatted at 2026-09-24 10:44:53 on dist-test-slave-6lwt"
server_key: "3b6b139c99e44f4ae17f08292eefcaf8"
server_key_iv: "f8119b4e8effd254b1e658bb7c7ce07f"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.171903 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal/instance:
uuid: "b6aecda326bd41458ca1e6eae80da49b"
format_stamp: "Formatted at 2026-09-24 10:44:53 on dist-test-slave-6lwt"
server_key: "3b6b139c99e44f4ae17f08292eefcaf8"
server_key_iv: "f8119b4e8effd254b1e658bb7c7ce07f"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.175717 9453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:44:53.178231 9554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.179448 9453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:53.179574 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "b6aecda326bd41458ca1e6eae80da49b"
format_stamp: "Formatted at 2026-09-24 10:44:53 on dist-test-slave-6lwt"
server_key: "3b6b139c99e44f4ae17f08292eefcaf8"
server_key_iv: "f8119b4e8effd254b1e658bb7c7ce07f"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.179679 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.194126 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.197405 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:53.197624 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.205621 9453 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.60:45581
I20260924 10:44:53.205907 9606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.60:45581 every 8 connection(s)
I20260924 10:44:53.206660 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
I20260924 10:44:53.209568 9607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:53.209725 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9453
I20260924 10:44:53.213858 9607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.60@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:53.223344 9078 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
I20260924 10:44:53.224761 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.211343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.2.182.60:45581)
0924 10:44:53.211731 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.211746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.211780 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:53.212485 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.212503 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.212814 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.213205 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.213992 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.215030 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.216057 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.216641 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.216809 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.219516 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.219525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.219542 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.219578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.221455 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.221983 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.221989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.221995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.222135 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.222455 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.222459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.222461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.223011 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.223242 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.223494 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.223791 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260924 10:44:53.225454 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.215609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.2.182.62:32999)
0924 10:44:53.215735 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.215739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.215957 (+ 218us) server_negotiation.cc:407] Connection header received
0924 10:44:53.216074 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.216077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.216131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.216213 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.216882 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.217596 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.218622 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.218832 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.218912 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.221304 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.221308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.221311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.221342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.223036 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.223556 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.223559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.223561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.223646 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.224498 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.224501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.224503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.224729 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.224862 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.225242 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.225349 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:44:53.225636 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.215449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33086)
0924 10:44:53.215817 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.215865 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.215979 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.216264 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.216269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.216576 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.216742 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.216747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.217740 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.217743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.218501 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.218511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.218989 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.218996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.219126 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.219729 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.219741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.221161 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.223149 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.223157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.223162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.223432 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.223761 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.223764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.223766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.223843 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.224856 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.224862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.225004 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.225151 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.225379 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":28,"thread_start_us":145,"threads_started":1}
I20260924 10:44:53.226652 9607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.60@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:53.235160 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.227139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44102)
0924 10:44:53.227270 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.227286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.227361 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.227742 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.227744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.227868 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.228289 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.228295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.229326 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.229329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.229953 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.229961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.230451 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.230454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.230558 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.230985 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.230995 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.232154 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.233726 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.233732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.233734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.233925 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.234220 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.234222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.234224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.234312 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.234672 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.234675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.234771 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.234877 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.235005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:44:53.235204 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.227271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.2.182.61:39393)
0924 10:44:53.227543 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.227547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.227558 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:44:53.227594 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.227596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.227633 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.227707 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.228408 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.229177 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.230069 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.230334 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.230435 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.232304 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.232307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.232310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.232339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.233622 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.234049 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.234051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.234052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.234122 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.234402 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.234407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.234409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.234577 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.234682 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.234937 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.235056 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260924 10:44:53.235909 9607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.60@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
I20260924 10:44:53.243379 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.236398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.2.182.60:45581)
0924 10:44:53.236506 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.236509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.236523 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:44:53.236555 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.236558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.236601 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.236661 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.237079 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.237551 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.238378 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.238589 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.238660 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.240386 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.240389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.240391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.240410 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.241838 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.242251 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.242257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.242259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.242330 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.242600 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.242603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.242605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.242764 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.242862 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.243117 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.243258 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":46}
I20260924 10:44:53.243378 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.236303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:48570)
0924 10:44:53.236409 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.236421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.236477 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.236683 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.236685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.236888 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.236988 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.236993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.237647 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.237650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.238240 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.238249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.238675 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.238677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.238742 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.239109 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.239120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.240271 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.241963 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.241968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.241973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.242158 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.242419 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.242424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.242426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.242509 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.242862 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.242865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.242952 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.243065 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.243203 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:44:53.246277 9607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Bootstrap starting.
I20260924 10:44:53.248510 9607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:53.249296 9607 log.cc:824] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:53.251063 9607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: No bootstrap required, opened a new log
I20260924 10:44:53.254242 9607 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } }
I20260924 10:44:53.254472 9607 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:53.254544 9607 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aecda326bd41458ca1e6eae80da49b, State: Initialized, Role: FOLLOWER
I20260924 10:44:53.255048 9607 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } }
I20260924 10:44:53.255484 9613 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } } }
I20260924 10:44:53.255616 9613 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:53.256211 9607 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:44:53.260782 9624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:53.260869 9624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:53.264025 9453 master_runner.cc:428] Detected that this master b6aecda326bd41458ca1e6eae80da49b is joining an existing cluster
I20260924 10:44:53.264093 9453 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.182.60:45581
I20260924 10:44:53.275668 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.266405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44104)
0924 10:44:53.266907 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.266922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.267301 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.267675 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.267678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.268000 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.268208 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.268216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269338 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.269341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.270698 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.270706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.271650 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.271653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.271723 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.272130 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.272140 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.272489 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.274348 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.274351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.274352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.274544 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.274819 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.274821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.274822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.274893 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.275271 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.275273 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.275344 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.275421 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.275532 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":119,"thread_start_us":98,"threads_started":1}
I20260924 10:44:53.275779 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.266801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.2.182.61:39393)
0924 10:44:53.266915 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.266919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.267000 (+ 81us) server_negotiation.cc:407] Connection header received
0924 10:44:53.267407 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.267411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.267479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.267552 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.268345 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269197 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.270807 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.271149 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.271222 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.272630 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.272634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.272637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.272662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.274248 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.274635 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.274638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.274640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.274728 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.274979 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.274982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.274984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.275183 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.275286 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.275500 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.275626 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:44:53.278111 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8949
I20260924 10:44:53.284997 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:43595
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:32999
--webserver_port=44031
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:53.294030 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.266566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33102)
0924 10:44:53.267149 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.267162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.267422 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.267766 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.267768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.268043 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.268214 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.268219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269338 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.269341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.270478 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.270488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.271151 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.271155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.271248 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.289433 (+ 18185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.289467 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.289945 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.292218 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.292224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.292562 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.292877 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.292882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.293002 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.293479 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.293482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.293586 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.293681 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.293831 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":125,"thread_start_us":105,"threads_started":1}
I20260924 10:44:53.294273 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.266927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.2.182.60:45581)
0924 10:44:53.267037 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.267041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.267399 (+ 358us) server_negotiation.cc:407] Connection header received
0924 10:44:53.267561 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.267564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.267600 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.267646 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.268439 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269210 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.270411 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.270765 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.270835 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.289593 (+ 18758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.289598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.289600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.289624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.291208 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292756 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.292761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.292765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.292851 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.293202 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.293207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.293211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.293475 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.293561 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.293921 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.294052 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":44}
I20260924 10:44:53.294925 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.266476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.2.182.62:32999)
0924 10:44:53.266621 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.266625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.267229 (+ 604us) server_negotiation.cc:407] Connection header received
0924 10:44:53.267527 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.267532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.267582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.267631 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.268344 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269197 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.270610 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.270928 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.271006 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.290152 (+ 19146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.290158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.290163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.290201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.292092 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292687 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.292691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.292693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.292779 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.293104 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.293107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.293109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.293338 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.293437 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.294654 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.294776 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:44:53.295018 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.267036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:48576)
0924 10:44:53.267320 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.267334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.267468 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.267815 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.267818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.268154 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.268306 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.268316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.269333 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.269336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.270291 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.270301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.270991 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.270995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.271083 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.271569 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.271585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.289430 (+ 17845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.292250 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.292261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.292636 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.292995 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.292999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.293002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.293097 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.293631 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.293633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.293739 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.294610 (+ 871us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.294775 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":43,"thread_start_us":96,"threads_started":1}
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246691, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.62@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
I20260924 10:44:53.308544 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.299253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:48584)
0924 10:44:53.299360 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.299374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.299451 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.299711 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.299714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.299900 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.300005 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.300011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.301060 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.301063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.301950 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.301960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.302546 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.302550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.302645 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.303145 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.303160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.304709 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.306798 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.306802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.306804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.307075 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.307365 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.307368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.307370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.307491 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.307929 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.307932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.308032 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.308159 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.308295 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 10:44:53.309250 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.299323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.2.182.60:45581)
0924 10:44:53.299421 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.299425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.299438 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:53.299544 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.299547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.299594 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.299675 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.300114 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.300933 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.302073 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.302414 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.302539 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.304853 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.304857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.304859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.304884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.306685 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.307181 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.307184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.307186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.307269 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.307620 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.307624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.307625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.307830 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.307925 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.308984 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.309098 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:44:53.310127 8887 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.182.60:45581: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b"
member_type: NON_VOTER
last_known_addr {
host: "127.2.182.60"
port: 45581
}
attrs {
promote: true
}
}
dest_uuid: "4c41f027916c4d61af007de8145c955c"
cas_config_opid_index: 5
I20260924 10:44:53.310734 8887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } }
I20260924 10:44:53.311746 9450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6aecda326bd41458ca1e6eae80da49b (127.2.182.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.312989 9581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:53.313380 9450 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.182.60:45581
I20260924 10:44:53.313551 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.313639 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.314373 9581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:44:53.314379 9453 master.cc:537] Master@127.2.182.60:45581 shutting down...
I20260924 10:44:53.315132 9448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:44:53.317400 8874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c -> Peer b6aecda326bd41458ca1e6eae80da49b (127.2.182.60:45581): Couldn't send request to peer b6aecda326bd41458ca1e6eae80da49b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:44:53.319404 9453 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:53.319653 9453 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:53.319742 9453 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: stopping tablet replica
I20260924 10:44:53.320885 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.311895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:44112)
0924 10:44:53.314035 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.314049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.314117 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.315552 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.315555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.315573 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:53.315654 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.315659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.316691 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.316695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.319333 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.319345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.320479 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.320483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.320593 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.320637 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.320707 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.320769 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071}
I20260924 10:44:53.321311 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.312126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.2.182.61:39393)
0924 10:44:53.315116 (+ 2990us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.315121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.315136 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:53.315185 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.315188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.315253 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.315344 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:53.315774 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.316542 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.319478 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.320104 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.320200 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.320330 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.321128 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.321202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2886}
I20260924 10:44:53.322127 9499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 LEARNER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:44:53.322531 9448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 10:44:53.324358 9499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6aecda326bd41458ca1e6eae80da49b (127.2.182.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.326210 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c41f027916c4d61af007de8145c955c. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.326311 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: LEARNER
I20260924 10:44:53.326381 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: NON_VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: true } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.326453 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: LEARNER
I20260924 10:44:53.326608 9448 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: attempting to promote NON_VOTER a5acbcf659a64f5ea7e8858d307bb03f to VOTER
I20260924 10:44:53.327046 9448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } }
I20260924 10:44:53.328231 9499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 LEARNER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 10:44:53.328370 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.328077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:48600)
0924 10:44:53.328227 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.328306 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 10:44:53.328979 9447 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 10:44:53.329073 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.328793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:48614)
0924 10:44:53.328941 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.328988 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 10:44:53.329150 8874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c -> Peer b6aecda326bd41458ca1e6eae80da49b (127.2.182.60:45581): Couldn't send request to peer b6aecda326bd41458ca1e6eae80da49b. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:44:53.330567 9449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.330988 9499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.331991 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.332084 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.332191 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.332263 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.332643 9635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.332800 9635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:53.338128 9453 master.cc:552] Master@127.2.182.60:45581 shutdown complete.
I20260924 10:44:53.338184 9453 master_runner.cc:305] Clearing existing system tablet
I20260924 10:44:53.338415 9453 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.338482 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.339798 9453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 10:44:53.340689 9637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.340946 9453 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.341065 9453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:53.341097 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "b6aecda326bd41458ca1e6eae80da49b"
format_stamp: "Formatted at 2026-09-24 10:44:53 on dist-test-slave-6lwt"
server_key: "3b6b139c99e44f4ae17f08292eefcaf8"
server_key_iv: "f8119b4e8effd254b1e658bb7c7ce07f"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.343067 9453 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:53.345287 9453 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:44:53.345374 9453 log.cc:1196] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 10:44:53.345618 9453 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Deleting consensus metadata
I20260924 10:44:53.345773 9453 master_runner.cc:315] Copying system tablet from 127.2.182.62:32999
I20260924 10:44:53.346849 9453 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.62:32999
I20260924 10:44:53.355829 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.347442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.2.182.62:32999)
0924 10:44:53.347553 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.347557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.347760 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:44:53.347936 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.347939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.347985 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.348035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.348671 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.349566 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.350673 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.350977 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.351042 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.352374 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.352378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.352381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.352406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.354105 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.354579 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.354583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.354585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.354666 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.354944 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.354948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.354950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.355152 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.355244 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.355563 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.355678 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:44:53.356366 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.347349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:33110)
0924 10:44:53.347678 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.347692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.347830 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.348185 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.348188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.348398 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.348538 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.348544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.349695 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.349698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.350546 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.350557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.351184 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.351187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.351273 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.351809 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.351822 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.352228 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.354208 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.354211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.354214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.354480 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.354760 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.354763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.354765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.354851 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.355236 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.355239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.355365 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.355969 (+ 604us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.356141 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":83,"threads_started":1}
I20260924 10:44:53.356726 8907 tablet_copy_service.cc:140] P 4c41f027916c4d61af007de8145c955c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b6aecda326bd41458ca1e6eae80da49b ({username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:33110)
I20260924 10:44:53.356837 8907 tablet_copy_service.cc:161] P 4c41f027916c4d61af007de8145c955c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b6aecda326bd41458ca1e6eae80da49b at {username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:33110: session id = b6aecda326bd41458ca1e6eae80da49b-00000000000000000000000000000000
I20260924 10:44:53.357903 8907 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:44:53.359236 9453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:44:53.361615 9453 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: tablet copy: Starting download of 0 data blocks...
I20260924 10:44:53.361785 9453 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: tablet copy: Starting download of 1 WAL segments...
I20260924 10:44:53.363183 9453 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:44:53.364753 8907 tablet_copy_service.cc:342] P 4c41f027916c4d61af007de8145c955c: Request end of tablet copy session b6aecda326bd41458ca1e6eae80da49b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:33110
I20260924 10:44:53.364832 8907 tablet_copy_service.cc:433] P 4c41f027916c4d61af007de8145c955c: ending tablet copy session b6aecda326bd41458ca1e6eae80da49b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b6aecda326bd41458ca1e6eae80da49b
I20260924 10:44:53.372814 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:53.374476 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.374533 9647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.374794 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.375046 9453 server_base.cc:1034] running on GCE node
I20260924 10:44:53.375188 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.375496 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.376644 9453 hybrid_clock.cc:648] HybridClock initialized: now 1790246693376621 us; error 42 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.378307 9453 init.cc:377] Logged in from keytab as oryx/127.2.182.60@KRBTEST.COM (short username oryx)
I20260924 10:44:53.378772 9453 webserver.cc:460] Webserver started at http://127.2.182.60:35593/ using document root <none> and password file <none>
I20260924 10:44:53.378948 9453 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.378990 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.380375 9453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:53.381435 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.381687 9453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:53.381757 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "b6aecda326bd41458ca1e6eae80da49b"
format_stamp: "Formatted at 2026-09-24 10:44:53 on dist-test-slave-6lwt"
server_key: "3b6b139c99e44f4ae17f08292eefcaf8"
server_key_iv: "f8119b4e8effd254b1e658bb7c7ce07f"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.381832 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.401008 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.403301 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.408762 9453 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.60:45581
I20260924 10:44:53.409426 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
I20260924 10:44:53.409942 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.60:45581 every 8 connection(s)
I20260924 10:44:53.411430 9707 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:44:53.412523 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Bootstrap starting.
I20260924 10:44:53.424032 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:44:53.424340 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Bootstrap complete.
I20260924 10:44:53.424921 9707 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } }
I20260924 10:44:53.425060 9707 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6aecda326bd41458ca1e6eae80da49b, State: Initialized, Role: LEARNER
I20260924 10:44:53.425160 9707 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } }
I20260924 10:44:53.425477 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.425583 9707 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:53.425674 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: LEARNER
W20260924 10:44:53.426244 9633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:53.426631 9633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:53.426715 9633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:44:53.430040 9724 catalog_manager.cc:1259] Loaded cluster ID: c9e4592b017045d2a59c0acecdbdfe92
I20260924 10:44:53.430155 9724 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: loading cluster ID for follower catalog manager: success
W20260924 10:44:53.431720 9633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:53.431802 9633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:53.431911 9633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:53.432382 9724 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: acquiring CA information for follower catalog manager: success
I20260924 10:44:53.433758 9724 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:44:53.437870 9633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:43595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=44031
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:53.438900 9633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:53.439905 9633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:53.450166 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.450326 9633 server_base.cc:1034] running on GCE node
W20260924 10:44:53.450237 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.450515 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.450917 9633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.451550 9633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.452718 9633 hybrid_clock.cc:648] HybridClock initialized: now 1790246693452703 us; error 28 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.455793 9633 init.cc:377] Logged in from keytab as oryx/127.2.182.1@KRBTEST.COM (short username oryx)
I20260924 10:44:53.456553 9633 webserver.cc:460] Webserver started at http://127.2.182.1:44031/ using document root <none> and password file <none>
I20260924 10:44:53.457134 9633 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.457207 9633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.460764 9633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:53.462772 9736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.464022 9633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:53.464118 9633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "e3ace47fb8494fe7addfe54733d314dc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "9c42db551faa4415ba344b48410cc86a"
server_key_iv: "161d495376824d195085911c8d1a0678"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.464614 9633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.470727 9212 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
I20260924 10:44:53.487923 9633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.491027 9633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:53.491223 9633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.491835 9633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:53.492671 9633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:53.492717 9633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.492803 9633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:53.492835 9633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.501569 9633 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:43595
I20260924 10:44:53.501646 9849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:43595 every 8 connection(s)
I20260924 10:44:53.502557 9633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:53.511684 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9633
I20260924 10:44:53.511921 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9083
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:53.519582 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.504423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:38467 (local address 127.2.182.62:32999)
0924 10:44:53.504562 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.504567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.505693 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 10:44:53.506722 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.506725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.506786 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.506843 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.508542 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509371 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.510436 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.510744 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.510846 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.515095 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.515098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.515101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.515128 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.517808 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518276 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.518281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.518286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.518361 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518665 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.518669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.518670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.518880 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.518964 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.519300 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.519420 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:44:53.519582 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.504367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:55073 (local address 127.2.182.61:39393)
0924 10:44:53.504520 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.504525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.505518 (+ 993us) server_negotiation.cc:407] Connection header received
0924 10:44:53.506692 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.506696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.506760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.506848 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.508537 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509131 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.510524 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.510868 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.510968 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.515095 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.515099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.515101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.515128 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.517033 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517691 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.517695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.517697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.517789 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518229 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.518235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.518237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.518497 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.518568 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.519300 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.519420 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:44:53.520650 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:38355
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:32999
--webserver_port=39067
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:53.520676 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.505892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.2.182.1:53895)
0924 10:44:53.506151 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.506166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.506525 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.507086 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.507088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.507702 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.508366 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.508384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509744 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.509747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.510586 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.510592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.511248 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.511252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.511360 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.512245 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.512263 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.514910 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.517449 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.517461 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.517718 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.518036 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.518047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.518149 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.518655 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.518660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.519024 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.519227 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.519596 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":97,"spinlock_wait_cycles":15232,"thread_start_us":86,"threads_started":1}
I20260924 10:44:53.520674 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.504775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.2.182.1:55073)
0924 10:44:53.505357 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.505408 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.506489 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.507014 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.507021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.507690 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.508375 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.508384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509265 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.509269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.510420 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.510429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.511006 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.511010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.511248 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.512522 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.512536 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.514902 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.517171 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.517197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.517562 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.517908 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.517915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.518106 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.518650 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.518656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.519027 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.520134 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.520251 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":250,"spinlock_wait_cycles":59776,"thread_start_us":569,"threads_started":1}
I20260924 10:44:53.521301 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.504775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:38467)
0924 10:44:53.505598 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.505612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.506525 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.507083 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.507086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.507711 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.508367 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.508382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509495 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.509498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.510317 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.510325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.511006 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.511010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.511248 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.512038 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.512063 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.514902 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.517921 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.517925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.518171 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.518474 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.518478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.518563 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.519037 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.519040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.519125 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.519794 (+ 669us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.519973 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":200,"spinlock_wait_cycles":19968,"thread_start_us":162,"threads_started":1}
I20260924 10:44:53.521471 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.506024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:53895 (local address 127.2.182.60:45581)
0924 10:44:53.506267 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.506272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.506287 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:53.506793 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.506796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.506875 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.506956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.508526 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.509616 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.510690 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.510998 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.511104 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.515091 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.515098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.515104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.515139 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.517324 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.517840 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.517846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.517851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.517935 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.518254 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.518260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.518265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.518515 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.518610 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.521094 (+ 2484us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.521238 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260924 10:44:53.522464 9851 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:53.522511 9852 heartbeater.cc:347] Connected to a master server at 127.2.182.61:39393
I20260924 10:44:53.522547 9850 heartbeater.cc:347] Connected to a master server at 127.2.182.60:45581
I20260924 10:44:53.522680 9850 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.522680 9852 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.522679 9851 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.523267 9850 heartbeater.cc:511] Master 127.2.182.60:45581 requested a full tablet report, sending...
I20260924 10:44:53.523267 9852 heartbeater.cc:511] Master 127.2.182.61:39393 requested a full tablet report, sending...
I20260924 10:44:53.523909 9851 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:53.524828 9481 ts_manager.cc:194] Registered new tserver with Master: e3ace47fb8494fe7addfe54733d314dc (127.2.182.1:43595)
I20260924 10:44:53.524827 9671 ts_manager.cc:194] Registered new tserver with Master: e3ace47fb8494fe7addfe54733d314dc (127.2.182.1:43595)
I20260924 10:44:53.525635 8887 ts_manager.cc:194] Re-registered known tserver with Master: e3ace47fb8494fe7addfe54733d314dc (127.2.182.1:43595)
I20260924 10:44:53.526288 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:38467
W20260924 10:44:53.632639 9858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:53.632879 9858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:53.632972 9858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:53.636958 9858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:53.637012 9858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:53.637068 9858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:53.642282 9858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:38355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=39067
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:53.643296 9858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:53.644272 9858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:53.653741 9858 server_base.cc:1034] running on GCE node
W20260924 10:44:53.653676 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.653676 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.653669 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.654385 9858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.655066 9858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.656265 9858 hybrid_clock.cc:648] HybridClock initialized: now 1790246693656221 us; error 54 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.658951 9858 init.cc:377] Logged in from keytab as oryx/127.2.182.2@KRBTEST.COM (short username oryx)
I20260924 10:44:53.659803 9858 webserver.cc:460] Webserver started at http://127.2.182.2:39067/ using document root <none> and password file <none>
I20260924 10:44:53.660348 9858 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.660455 9858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.664129 9858 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:53.666188 9873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.667297 9858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:53.667433 9858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "a54411e80dac479ba742837550cdedcc"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "4a5b4e596cd2cb454adfe18b090f5d72"
server_key_iv: "cfb539beec0608235b55e78147b4a541"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.667948 9858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.679168 9346 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
I20260924 10:44:53.686743 9858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.689713 9858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:53.689906 9858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.690610 9858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:53.691462 9858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:53.691533 9858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.691601 9858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:53.691648 9858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.700743 9858 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:38355
I20260924 10:44:53.700759 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:38355 every 8 connection(s)
I20260924 10:44:53.701769 9858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:53.709473 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9858
I20260924 10:44:53.709647 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9217
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
I20260924 10:44:53.717427 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:38709
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:32999
--webserver_port=37131
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:53.718096 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.703626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:41005 (local address 127.2.182.62:32999)
0924 10:44:53.703754 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.703758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.704687 (+ 929us) server_negotiation.cc:407] Connection header received
0924 10:44:53.705838 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.705841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.705894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.705958 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.707312 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.708137 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.709572 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.709867 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.709931 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.713457 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.713463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.713469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.713503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.715497 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716177 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.716181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.716183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.716415 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716825 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.716829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.716830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.717054 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.717138 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.717825 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.717935 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:53.719645 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.704014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:41005)
0924 10:44:53.704517 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.704568 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.705624 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.706125 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.706132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.706654 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.707159 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.707172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.708264 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.708267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.709440 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.709450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.710757 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.710763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.710922 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.711464 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.711484 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.713272 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.715598 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.715601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.715603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.716062 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.716589 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.716596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.716714 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.717192 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.717198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.717550 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.718232 (+ 682us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.718397 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":299,"spinlock_wait_cycles":14592,"thread_start_us":563,"threads_started":1}
I20260924 10:44:53.719668 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.703635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:43185 (local address 127.2.182.61:39393)
0924 10:44:53.703761 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.703765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.704955 (+ 1190us) server_negotiation.cc:407] Connection header received
0924 10:44:53.705816 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.705820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.705873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.705951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.707293 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.707970 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.709357 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.709671 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.709778 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.713111 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.713115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.713118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.713146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.715413 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716177 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.716181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.716183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.716277 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716833 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.716837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.716838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.717063 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.717140 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.718313 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.718420 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:44:53.720093 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.705206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:50675 (local address 127.2.182.60:45581)
0924 10:44:53.705346 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.705350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.705403 (+ 53us) server_negotiation.cc:407] Connection header received
0924 10:44:53.705837 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.705841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.705894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.705947 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.707312 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.708124 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.709669 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.710008 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.710088 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.716014 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.716021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.716027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.716063 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.717851 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.718621 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.718628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.718632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.718723 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.719042 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.719048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.719051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.719292 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.719369 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.719741 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.719884 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:44:53.720764 9988 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:53.720847 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.704014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.2.182.2:43185)
0924 10:44:53.704833 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.704847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.705619 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.706123 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.706130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.706609 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.707156 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.707168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.708099 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.708107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.709213 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.709226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.710514 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.710517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.710648 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.711298 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.711323 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.712899 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.715547 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.715556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.715575 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.716062 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.716537 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.716543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.716545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.716714 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.717192 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.717198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.717550 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.717746 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.718056 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":165,"spinlock_wait_cycles":51968,"thread_start_us":157,"threads_started":1}
I20260924 10:44:53.721042 9988 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.721413 9989 heartbeater.cc:347] Connected to a master server at 127.2.182.61:39393
I20260924 10:44:53.721493 9989 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.721606 9988 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:53.721732 9989 heartbeater.cc:511] Master 127.2.182.61:39393 requested a full tablet report, sending...
I20260924 10:44:53.722694 8887 ts_manager.cc:194] Re-registered known tserver with Master: a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
I20260924 10:44:53.722867 9481 ts_manager.cc:194] Registered new tserver with Master: a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
I20260924 10:44:53.723318 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:41005
I20260924 10:44:53.721444 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.705071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.2.182.2:50675)
0924 10:44:53.705340 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.705351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.705624 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.706125 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.706132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.706654 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.707159 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.707172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.708264 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.708267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.709440 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.709456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.713586 (+ 4130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.713593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.713722 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.714202 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.714221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.715841 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.718126 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.718132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.718238 (+ 106us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.718507 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.718851 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.718854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.718859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.718936 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.719447 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.719450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.719559 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.720590 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.720755 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":240,"spinlock_wait_cycles":21632,"thread_start_us":101,"threads_started":1}
I20260924 10:44:53.724447 9987 heartbeater.cc:347] Connected to a master server at 127.2.182.60:45581
I20260924 10:44:53.724535 9987 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.725025 9987 heartbeater.cc:511] Master 127.2.182.60:45581 requested a full tablet report, sending...
I20260924 10:44:53.725857 9671 ts_manager.cc:194] Registered new tserver with Master: a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
W20260924 10:44:53.833660 9994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:53.833902 9994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:53.833938 9994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:53.838025 9994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:53.838078 9994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:53.838132 9994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:44:53.842986 9994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:38709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=37131
--tserver_master_addrs=127.2.182.62:32999,127.2.182.61:39393,127.2.182.60:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:53.843888 9994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:53.844779 9994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:53.853772 10002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.853785 9999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:53.853809 10000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:53.853940 9994 server_base.cc:1034] running on GCE node
I20260924 10:44:53.854609 9994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:53.855218 9994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:53.856386 9994 hybrid_clock.cc:648] HybridClock initialized: now 1790246693856366 us; error 34 us; skew 500 ppm
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:53.859153 9994 init.cc:377] Logged in from keytab as oryx/127.2.182.3@KRBTEST.COM (short username oryx)
I20260924 10:44:53.859988 9994 webserver.cc:460] Webserver started at http://127.2.182.3:37131/ using document root <none> and password file <none>
I20260924 10:44:53.860553 9994 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:53.860646 9994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:53.864423 9994 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:44:53.866431 10009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.867619 9994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:53.867754 9994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "04be2753ef0b4fd79f23ab480a3c141a"
format_stamp: "Formatted at 2026-09-24 10:44:52 on dist-test-slave-6lwt"
server_key: "c2702ecb571f6e1b6a032f299e25bc6e"
server_key_iv: "48a0dda20436bf5972ede1d8e4949b07"
server_key_version: "encryptionkey@0"
I20260924 10:44:53.868299 9994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:53.899797 9994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:53.903182 9994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:53.903384 9994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:53.903980 9994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:44:53.904968 9994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:53.905030 9994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.905087 9994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:53.905109 9994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:53.913909 9994 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38709
I20260924 10:44:53.913938 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:38709 every 8 connection(s)
I20260924 10:44:53.914951 9994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:44:53.924860 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 9994
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.60@KRBTEST.COM
Sep 24 10:44:53 dist-test-slave-6lwt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246693, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.61@KRBTEST.COM
I20260924 10:44:53.933960 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.916719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:51483 (local address 127.2.182.61:39393)
0924 10:44:53.916845 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.916849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.917794 (+ 945us) server_negotiation.cc:407] Connection header received
0924 10:44:53.919004 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.919007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.919056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.919109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.920381 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.921211 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.922641 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.922993 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.923068 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.929175 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.929179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.929182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.929211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.931288 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.931936 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.931941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.931945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.932030 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.932427 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.932430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.932432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.932677 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.932768 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.933687 (+ 919us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.933808 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:53.934943 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.917205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:57517)
0924 10:44:53.917642 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.917695 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.918777 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.919284 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.919291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.919732 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.920219 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.920232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.921103 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.921107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.922504 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.922514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.923223 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.923229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.923368 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.924221 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.924246 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.928964 (+ 4718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.931863 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.931869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.931873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.932135 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.932426 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.932428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.932430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.932511 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.932985 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.932988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.933206 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.933435 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.933816 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":182,"spinlock_wait_cycles":19584,"thread_start_us":127,"threads_started":1}
I20260924 10:44:53.935607 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.916738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:57517 (local address 127.2.182.62:32999)
0924 10:44:53.916881 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.916903 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.917794 (+ 891us) server_negotiation.cc:407] Connection header received
0924 10:44:53.918987 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.918990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.919052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.919127 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.920370 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.920960 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.922642 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.922992 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.923068 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.929176 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.929179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.929182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.929211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.931730 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.932240 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.932243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.932245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.932321 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.932614 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.932617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.932619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.932830 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.932933 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.935355 (+ 2422us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.935467 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:44:53.936393 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.917205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.2.182.3:51483)
0924 10:44:53.917643 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.917695 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.918778 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.919285 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.919292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.919771 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.920231 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.920237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.921438 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.921443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.922504 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.922514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.923237 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.923241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.923940 (+ 699us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.925653 (+ 1713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.925671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.928964 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.931425 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.931434 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.931468 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.931826 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.932140 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.932142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.932144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.932322 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.932821 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.932828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.933206 (+ 378us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.933435 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.935881 (+ 2446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":2928,"spinlock_wait_cycles":115328,"thread_start_us":158,"threads_started":1}
I20260924 10:44:53.937130 10124 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32999
I20260924 10:44:53.937355 10124 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.937871 10124 heartbeater.cc:511] Master 127.2.182.62:32999 requested a full tablet report, sending...
I20260924 10:44:53.938349 10125 heartbeater.cc:347] Connected to a master server at 127.2.182.61:39393
I20260924 10:44:53.938411 10125 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.938452 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.918346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:47327 (local address 127.2.182.60:45581)
0924 10:44:53.918464 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.918467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.918541 (+ 74us) server_negotiation.cc:407] Connection header received
0924 10:44:53.918999 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.919003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.919056 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.919109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.920382 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.921211 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.922522 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.922849 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.922928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.929183 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.929187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.929190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.929220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.936481 (+ 7261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.937047 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.937054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.937059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.937157 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.937487 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.937492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.937495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.937710 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.937808 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.938180 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.938305 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:44:53.938548 10125 heartbeater.cc:511] Master 127.2.182.61:39393 requested a full tablet report, sending...
I20260924 10:44:53.939276 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.918189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.2.182.3:47327)
0924 10:44:53.918443 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.918465 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.918778 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.919285 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.919292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.919771 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:53.920228 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.920237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.921324 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.921328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.922369 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.922380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.923149 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.923152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.923395 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.925533 (+ 2138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:53.925553 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:53.929022 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:53.936588 (+ 7566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.936591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.936593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.936882 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.937271 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:53.937275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:53.937276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:53.937367 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:53.937902 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:53.937905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:53.938011 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.938578 (+ 567us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.939052 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":186,"spinlock_wait_cycles":12928,"thread_start_us":96,"threads_started":1}
I20260924 10:44:53.939320 9481 ts_manager.cc:194] Registered new tserver with Master: 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:53.940609 8887 ts_manager.cc:194] Re-registered known tserver with Master: 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:53.941437 8887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:57517
I20260924 10:44:53.941668 10123 heartbeater.cc:347] Connected to a master server at 127.2.182.60:45581
I20260924 10:44:53.941751 10123 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:53.941903 10123 heartbeater.cc:511] Master 127.2.182.60:45581 requested a full tablet report, sending...
I20260924 10:44:53.942724 9671 ts_manager.cc:194] Registered new tserver with Master: 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:53.944788 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.931041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:36738)
0924 10:44:53.931494 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.931507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.931580 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.932053 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.932056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.932072 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:53.932175 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.932180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.933196 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.933200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.943304 (+ 10104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.943314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.944405 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.944409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.944502 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.944523 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.944589 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.944664 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":61,"threads_started":1}
I20260924 10:44:53.945415 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.931299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.2.182.60:45581)
0924 10:44:53.931742 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.931745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.931756 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:44:53.931794 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.931796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.931860 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.931963 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:53.932277 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.933076 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.943433 (+ 10357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.944114 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.944208 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.944346 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.945022 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.945096 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":57,"threads_started":1}
I20260924 10:44:53.945755 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.927768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.2.182.60:45581)
0924 10:44:53.927996 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.927999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.929185 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 10:44:53.929218 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.929220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.929263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.929348 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.930364 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.935567 (+ 5203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.937845 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.938240 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.939979 (+ 1739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.940095 (+ 116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.940098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.940100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.940113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.942847 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.943373 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.943376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.943378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.943458 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.943753 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.943755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.943757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.943986 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.944632 (+ 646us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.944711 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.944831 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260924 10:44:53.945904 9681 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 10:44:53.946413 10142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c41f027916c4d61af007de8145c955c. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.946672 10142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: LEARNER
I20260924 10:44:53.948607 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.927096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.2.182.61:39393)
0924 10:44:53.934115 (+ 7019us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.934119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.934132 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:53.934165 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.934167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.934205 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.934287 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.935079 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.935828 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.941559 (+ 5731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.941879 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.941988 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.945030 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.945034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.945039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.945066 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.946815 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.947295 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.947298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.947300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.947385 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.947701 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.947705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.947709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.947943 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.948098 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.948327 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.948445 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6963,"thread_start_us":76,"threads_started":1}
I20260924 10:44:53.951781 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.927055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.2.182.62:32999)
0924 10:44:53.929669 (+ 2614us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.929673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.929685 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:53.929725 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.929729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.929780 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.941014 (+ 11234us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:53.942220 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.942908 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.946835 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.947146 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.947654 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.948680 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:53.948684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:53.948686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:53.948979 (+ 293us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:53.950498 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.950866 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.950868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.950869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.950939 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:53.951157 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:53.951159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:53.951160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:53.951321 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:53.951411 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.951573 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.951667 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2533,"thread_start_us":105,"threads_started":1}
I20260924 10:44:53.954849 8887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:53.958091 8887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:53.960981 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: NON_VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: true } } }
I20260924 10:44:53.961138 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.964193 10145 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: attempting to promote NON_VOTER b6aecda326bd41458ca1e6eae80da49b to VOTER
I20260924 10:44:53.964545 10145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:53.965858 9681 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 LEARNER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:44:53.966399 10145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:44:53.967765 9499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Refusing update from remote peer 4c41f027916c4d61af007de8145c955c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:44:53.968182 10145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b6aecda326bd41458ca1e6eae80da49b (127.2.182.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.968367 9450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:44:53.968986 9681 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b6aecda326bd41458ca1e6eae80da49b (127.2.182.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.969650 9499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b6aecda326bd41458ca1e6eae80da49b (127.2.182.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.970041 10145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.970191 10145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.970335 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.970410 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.971264 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.971395 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:53.971645 10142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.971767 10142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:53.971971 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c41f027916c4d61af007de8145c955c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:53.972103 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:53.979293 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.973991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:38709 (local address 127.0.0.1:55898)
0924 10:44:53.974174 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.974188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.974270 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.975085 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.975091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.975107 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:53.975217 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.975223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.976694 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.976698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.977770 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.977779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.978900 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.978904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.978993 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.979030 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.979078 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.979153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 10:44:53.980252 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.973988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.2.182.3:38709)
0924 10:44:53.974557 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.974570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.974595 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:44:53.974678 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.974686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.974923 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.975387 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:53.975536 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.976579 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.977918 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.978767 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.979712 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.979910 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.979967 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.980075 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":129,"threads_started":1}
I20260924 10:44:53.980389 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.974712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.2.182.2:38355)
0924 10:44:53.975050 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.975065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.975377 (+ 312us) server_negotiation.cc:407] Connection header received
0924 10:44:53.975493 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.975501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.975710 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.975888 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:53.976401 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.977560 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.979076 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.979829 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.979977 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.980125 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.980175 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.980263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260924 10:44:53.980499 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.974188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43595 (local address 127.0.0.1:48428)
0924 10:44:53.974933 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.974949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.975042 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.976049 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.976052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.976068 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:53.976222 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.976229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.977726 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.977730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.978678 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.978691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.980025 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.980029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.980121 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.980147 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.980239 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.980335 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":95,"thread_start_us":107,"threads_started":1}
I20260924 10:44:53.980528 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.974497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:38355 (local address 127.0.0.1:60634)
0924 10:44:53.975129 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:53.975143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:53.975256 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:53.976075 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:53.976078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:53.976106 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:53.976239 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.976245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.977707 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.977711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.978934 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:53.978942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.979969 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.979973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:53.980055 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.980077 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:44:53.980139 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:44:53.980208 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":71,"thread_start_us":110,"threads_started":1}
I20260924 10:44:53.981019 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:53.974597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.2.182.1:43595)
0924 10:44:53.975271 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:53.975287 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:53.975317 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:44:53.975547 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:53.975552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:53.975748 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:53.975915 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:53.976359 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.977587 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.978841 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:53.979731 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:53.979841 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:53.979960 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:53.980726 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 10:44:53.980831 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":66,"threads_started":1}
I20260924 10:44:53.983572 10057 tablet_service.cc:1467] Processing CreateTablet for tablet 62295c317d77488193ca8422ad6f2910 (DEFAULT_TABLE table=test-table [id=de1615f9e90648edbbc2627634c6aadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:53.983976 9921 tablet_service.cc:1467] Processing CreateTablet for tablet 62295c317d77488193ca8422ad6f2910 (DEFAULT_TABLE table=test-table [id=de1615f9e90648edbbc2627634c6aadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:53.983984 9784 tablet_service.cc:1467] Processing CreateTablet for tablet 62295c317d77488193ca8422ad6f2910 (DEFAULT_TABLE table=test-table [id=de1615f9e90648edbbc2627634c6aadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:53.984848 10057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62295c317d77488193ca8422ad6f2910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:53.985018 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62295c317d77488193ca8422ad6f2910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:53.985018 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62295c317d77488193ca8422ad6f2910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:53.992527 10159 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Bootstrap starting.
I20260924 10:44:53.993741 10161 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Bootstrap starting.
I20260924 10:44:53.993741 10160 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Bootstrap starting.
I20260924 10:44:53.995292 10159 tablet_bootstrap.cc:654] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:53.995903 10160 tablet_bootstrap.cc:654] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:53.995903 10161 tablet_bootstrap.cc:654] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:53.996140 10159 log.cc:824] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:53.996760 10161 log.cc:824] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:53.996760 10160 log.cc:824] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:53.998464 10159 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: No bootstrap required, opened a new log
I20260924 10:44:53.998697 10159 ts_tablet_manager.cc:1397] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:44:53.998735 10161 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: No bootstrap required, opened a new log
I20260924 10:44:53.998908 10161 ts_tablet_manager.cc:1397] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:44:53.999259 10160 tablet_bootstrap.cc:492] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: No bootstrap required, opened a new log
I20260924 10:44:53.999435 10160 ts_tablet_manager.cc:1397] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:44:54.002897 10159 raft_consensus.cc:348] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.003191 10159 raft_consensus.cc:374] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:54.003253 10159 raft_consensus.cc:729] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04be2753ef0b4fd79f23ab480a3c141a, State: Initialized, Role: FOLLOWER
I20260924 10:44:54.003595 10161 raft_consensus.cc:348] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.003747 10159 consensus_queue.cc:260] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.003851 10161 raft_consensus.cc:374] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:54.003914 10161 raft_consensus.cc:729] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54411e80dac479ba742837550cdedcc, State: Initialized, Role: FOLLOWER
I20260924 10:44:54.003921 10160 raft_consensus.cc:348] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.004169 10160 raft_consensus.cc:374] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:54.004226 10160 raft_consensus.cc:729] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ace47fb8494fe7addfe54733d314dc, State: Initialized, Role: FOLLOWER
I20260924 10:44:54.004635 10124 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
I20260924 10:44:54.004649 10160 consensus_queue.cc:260] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.004880 10159 ts_tablet_manager.cc:1428] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:44:54.005576 10160 ts_tablet_manager.cc:1428] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260924 10:44:54.005687 9851 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
I20260924 10:44:54.004374 10161 consensus_queue.cc:260] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.009693 10161 ts_tablet_manager.cc:1428] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20260924 10:44:54.010020 9988 heartbeater.cc:503] Master 127.2.182.62:32999 was elected leader, sending a full tablet report...
W20260924 10:44:54.166261 10126 tablet.cc:2369] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:54.166319 10166 raft_consensus.cc:482] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:54.166514 10166 raft_consensus.cc:504] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.167732 10166 leader_election.cc:288] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709), a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
I20260924 10:44:54.171670 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.168092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:51377 (local address 127.2.182.3:38709)
0924 10:44:54.168198 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.168205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.168284 (+ 79us) server_negotiation.cc:407] Connection header received
0924 10:44:54.168435 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.168440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.168488 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.168585 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:54.168924 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.169506 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.170559 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.171015 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.171123 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.171208 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.171467 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.171534 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:44:54.171710 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.168074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:48993 (local address 127.2.182.2:38355)
0924 10:44:54.168199 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.168204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.168392 (+ 188us) server_negotiation.cc:407] Connection header received
0924 10:44:54.168581 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.168584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.168634 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.168708 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:54.168960 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.169482 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.170506 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.170944 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.171059 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.171121 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.171436 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.171555 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:44:54.171728 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.167981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:38355 (local address 127.2.182.1:48993)
0924 10:44:54.168278 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:54.168289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:54.168424 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:54.168716 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:54.168718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:54.168731 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:54.168826 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:54.168830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.169583 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.169587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:54.170393 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:54.170402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.171087 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.171092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:54.171227 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.171278 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:44:54.171410 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:44:54.171514 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260924 10:44:54.172351 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.167895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:38709 (local address 127.2.182.1:51377)
0924 10:44:54.168191 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:54.168206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:54.168318 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:54.168595 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:54.168598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:54.168612 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:54.168781 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:54.168791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.169635 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.169638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:54.170420 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:54.170429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.171235 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.171238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:54.171323 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.171349 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:44:54.172181 (+ 832us) client_negotiation.cc:240] Negotiation successful
0924 10:44:54.172253 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":63,"thread_start_us":70,"threads_started":1}
I20260924 10:44:54.172369 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62295c317d77488193ca8422ad6f2910" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54411e80dac479ba742837550cdedcc" is_pre_election: true
I20260924 10:44:54.172662 9941 raft_consensus.cc:2454] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 0.
I20260924 10:44:54.173238 9739 leader_election.cc:302] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a54411e80dac479ba742837550cdedcc, e3ace47fb8494fe7addfe54733d314dc; no voters:
I20260924 10:44:54.173446 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62295c317d77488193ca8422ad6f2910" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04be2753ef0b4fd79f23ab480a3c141a" is_pre_election: true
I20260924 10:44:54.173501 10166 raft_consensus.cc:2781] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:54.173576 10166 raft_consensus.cc:482] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:54.173619 10166 raft_consensus.cc:3037] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:54.173746 10077 raft_consensus.cc:2454] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 0.
I20260924 10:44:54.174782 10166 raft_consensus.cc:504] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.175117 10166 leader_election.cc:288] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 election: Requested vote from peers 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709), a54411e80dac479ba742837550cdedcc (127.2.182.2:38355)
I20260924 10:44:54.175406 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62295c317d77488193ca8422ad6f2910" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04be2753ef0b4fd79f23ab480a3c141a"
I20260924 10:44:54.175535 10077 raft_consensus.cc:3037] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:54.175544 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62295c317d77488193ca8422ad6f2910" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54411e80dac479ba742837550cdedcc"
I20260924 10:44:54.175683 9941 raft_consensus.cc:3037] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:54.176587 10077 raft_consensus.cc:2454] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 1.
I20260924 10:44:54.176738 9941 raft_consensus.cc:2454] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 1.
I20260924 10:44:54.176883 9740 leader_election.cc:302] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04be2753ef0b4fd79f23ab480a3c141a, e3ace47fb8494fe7addfe54733d314dc; no voters:
I20260924 10:44:54.177075 10166 raft_consensus.cc:2781] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:54.177317 10166 raft_consensus.cc:686] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 1 LEADER]: Becoming Leader. State: Replica: e3ace47fb8494fe7addfe54733d314dc, State: Running, Role: LEADER
I20260924 10:44:54.177630 10166 consensus_queue.cc:237] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } }
I20260924 10:44:54.181064 8887 catalog_manager.cc:5495] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ace47fb8494fe7addfe54733d314dc (127.2.182.1). New cstate: current_term: 1 leader_uuid: "e3ace47fb8494fe7addfe54733d314dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } health_report { overall_health: UNKNOWN } } }
W20260924 10:44:54.203701 9990 tablet.cc:2369] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:54.212199 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.208588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.2.182.1:43595)
0924 10:44:54.208747 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.208753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.208789 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:44:54.209004 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.209007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.209071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.209179 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:54.209426 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.209927 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.210769 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.211008 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.211119 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.211420 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:54.211808 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:54.211933 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.212019 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.212070 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:44:54.217571 9941 raft_consensus.cc:1261] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Refusing update from remote peer e3ace47fb8494fe7addfe54733d314dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:54.217572 10077 raft_consensus.cc:1261] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Refusing update from remote peer e3ace47fb8494fe7addfe54733d314dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:54.218240 10170 consensus_queue.cc:1035] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:54.218415 10166 consensus_queue.cc:1035] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:54.229867 10173 mvcc.cc:205] Tried to move back new op lower bound from 7332850459508699136 to 7332850459354013696. Current Snapshot: MvccSnapshot[applied={T|T < 7332850459508699136}]
I20260924 10:44:54.238125 8887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:54.238284 8887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:54.242334 8887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c41f027916c4d61af007de8145c955c: Sending DeleteTablet for 3 replicas of tablet 62295c317d77488193ca8422ad6f2910
I20260924 10:44:54.243057 9784 tablet_service.cc:1514] Processing DeleteTablet for tablet 62295c317d77488193ca8422ad6f2910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:54 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:48428
I20260924 10:44:54.243389 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet 62295c317d77488193ca8422ad6f2910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:54 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:60634
I20260924 10:44:54.244056 10057 tablet_service.cc:1514] Processing DeleteTablet for tablet 62295c317d77488193ca8422ad6f2910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:54 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:55898
I20260924 10:44:54.244259 10184 tablet_replica.cc:330] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: stopping tablet replica
I20260924 10:44:54.244700 10184 raft_consensus.cc:2229] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:54.245064 10184 raft_consensus.cc:2258] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:54.245268 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 8855
I20260924 10:44:54.245791 10183 tablet_replica.cc:330] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: stopping tablet replica
I20260924 10:44:54.246100 10183 raft_consensus.cc:2229] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:54.246512 10183 raft_consensus.cc:2258] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:54.246913 10186 tablet_replica.cc:330] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: stopping tablet replica
I20260924 10:44:54.247251 10186 raft_consensus.cc:2229] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:54.247574 10186 raft_consensus.cc:2258] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:54.250218 10183 ts_tablet_manager.cc:1905] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:54.252481 10184 ts_tablet_manager.cc:1905] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:54.253257 10183 ts_tablet_manager.cc:1918] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:54.254195 10186 ts_tablet_manager.cc:1905] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:54.255698 10184 ts_tablet_manager.cc:1918] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:54.255795 10184 log.cc:1196] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/62295c317d77488193ca8422ad6f2910
I20260924 10:44:54.256131 10184 ts_tablet_manager.cc:1939] T 62295c317d77488193ca8422ad6f2910 P a54411e80dac479ba742837550cdedcc: Deleting consensus metadata
I20260924 10:44:54.256955 10186 ts_tablet_manager.cc:1918] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:54.257099 10183 log.cc:1196] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/62295c317d77488193ca8422ad6f2910
W20260924 10:44:54.257302 9877 connection.cc:441] server connection from 127.0.0.1:60634 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60634 (request call id 1) could send its response
I20260924 10:44:54.257512 10183 ts_tablet_manager.cc:1939] T 62295c317d77488193ca8422ad6f2910 P e3ace47fb8494fe7addfe54733d314dc: Deleting consensus metadata
I20260924 10:44:54.257618 10186 log.cc:1196] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/62295c317d77488193ca8422ad6f2910
I20260924 10:44:54.257932 10186 ts_tablet_manager.cc:1939] T 62295c317d77488193ca8422ad6f2910 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting consensus metadata
W20260924 10:44:54.258941 10013 connection.cc:441] server connection from 127.0.0.1:55898 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55898 (request call id 1) could send its response
W20260924 10:44:54.259757 9737 connection.cc:441] server connection from 127.0.0.1:48428 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48428 (request call id 1) could send its response
I20260924 10:44:54.265736 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.243996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.2.182.61:39393)
0924 10:44:54.244109 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.244114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.245381 (+ 1267us) server_negotiation.cc:407] Connection header received
0924 10:44:54.246071 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.246077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.246135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.246216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:54.247621 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.248491 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.249862 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.250289 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.250394 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.261013 (+ 10619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:54.261020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:54.261023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:54.261050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:54.262877 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.263382 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.263385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.263387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.263484 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.264338 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.264343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.264347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.264580 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:54.264713 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.265102 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.265528 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:54.269711 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.258523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.2.182.61:39393)
0924 10:44:54.258640 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.258645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.259364 (+ 719us) server_negotiation.cc:407] Connection header received
0924 10:44:54.259467 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.259472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.259526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.259603 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:54.261033 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.261791 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.263104 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.263476 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.263556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.265229 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:54.265234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:54.265237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:54.265263 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:54.267420 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.268298 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.268301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.268303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.268387 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.268753 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.268757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.268758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.268990 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:54.269092 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.269444 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.269567 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:54.269711 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.258056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.2.182.60:45581)
0924 10:44:54.258178 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.258184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.259021 (+ 837us) server_negotiation.cc:407] Connection header received
0924 10:44:54.259523 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.259527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.259582 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.259635 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:54.260824 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.261628 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.262794 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.263174 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.263248 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.264841 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:54.264845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:54.264848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:54.264873 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:54.267744 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.268305 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.268308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.268311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.268387 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.268753 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.268757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.268758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.269004 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:54.269092 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.269444 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.269578 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:44:54.274847 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:54.244411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.2.182.60:45581)
0924 10:44:54.244515 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:54.244519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:54.266241 (+ 21722us) server_negotiation.cc:407] Connection header received
0924 10:44:54.266401 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:54.266404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:54.266450 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:54.266526 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:54.267221 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.268036 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.269207 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:54.269540 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:54.269608 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:54.271613 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:54.271617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:54.271620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:54.271643 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:54.273346 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.273715 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.273718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.273720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.273803 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:54.274046 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:54.274049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:54.274051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:54.274240 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:54.274362 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:54.274582 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:44:54.274692 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 10:44:55.230458 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.230000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:53069)
0924 10:44:55.230238 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.230340 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
W20260924 10:44:55.230790 10124 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.231890 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.231342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:44989)
0924 10:44:55.231703 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.231788 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
W20260924 10:44:55.232366 9851 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.232967 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.232548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:48839)
0924 10:44:55.232757 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.232843 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
W20260924 10:44:55.233222 9988 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.742919 10197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c41f027916c4d61af007de8145c955c)
I20260924 10:44:55.743108 10198 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c41f027916c4d61af007de8145c955c)
I20260924 10:44:55.743178 10197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:55.743269 10198 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:55.743824 10198 leader_election.cc:288] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), b6aecda326bd41458ca1e6eae80da49b (127.2.182.60:45581)
I20260924 10:44:55.744275 10197 leader_election.cc:288] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61:39393)
I20260924 10:44:55.744740 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.743991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52938)
0924 10:44:55.744550 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.744638 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":65,"threads_started":1}
I20260924 10:44:55.744753 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.744384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52944)
0924 10:44:55.744616 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.744676 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
W20260924 10:44:55.744972 9476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
W20260924 10:44:55.745008 9658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.746073 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.745721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52946)
0924 10:44:55.745957 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.746006 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260924 10:44:55.746169 9658 leader_election.cc:334] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.746434 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.746063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52958)
0924 10:44:55.746305 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.746368 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260924 10:44:55.746590 9476 leader_election.cc:334] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.749370 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.744337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45581 (local address 127.0.0.1:36758)
0924 10:44:55.744918 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.744933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:55.745046 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:55.745483 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:55.745488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:55.745508 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:55.745618 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:55.745628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.746605 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.746611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:55.747896 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:55.747907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.748898 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748925 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:55.749020 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:55.749045 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:44:55.749131 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:44:55.749203 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":55,"threads_started":1}
I20260924 10:44:55.749370 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.744531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:39393 (local address 127.0.0.1:60586)
0924 10:44:55.745125 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.745136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:55.745247 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:55.745707 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:55.745711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:55.745731 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:55.745811 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:55.745816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.746800 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.746803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:55.748013 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:55.748025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.748946 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:55.749046 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:55.749068 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:44:55.749142 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:44:55.749203 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"spinlock_wait_cycles":5760,"thread_start_us":44,"threads_started":1}
I20260924 10:44:55.749400 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.744755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.2.182.61:39393)
0924 10:44:55.745480 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:55.745485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:55.745498 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:55.745546 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:55.745551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:55.745595 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:55.745685 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:55.745921 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.746659 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748228 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.748825 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748951 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:55.749002 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:55.749166 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 10:44:55.749238 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":65,"threads_started":1}
I20260924 10:44:55.749379 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.744602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.2.182.60:45581)
0924 10:44:55.745011 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:55.745017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:55.745036 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:44:55.745182 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:55.745188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:55.745265 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:55.745343 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:55.745749 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.746495 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748032 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:55.748753 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:55.748881 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:55.748982 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:55.749166 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:44:55.749238 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"spinlock_wait_cycles":9600,"thread_start_us":56,"threads_started":1}
I20260924 10:44:55.749977 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6aecda326bd41458ca1e6eae80da49b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" is_pre_election: true
I20260924 10:44:55.750049 9681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b6aecda326bd41458ca1e6eae80da49b" is_pre_election: true
I20260924 10:44:55.750197 9681 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5acbcf659a64f5ea7e8858d307bb03f in term 1.
I20260924 10:44:55.750197 9499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6aecda326bd41458ca1e6eae80da49b in term 1.
I20260924 10:44:55.750524 9476 leader_election.cc:302] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f, b6aecda326bd41458ca1e6eae80da49b; no voters: 4c41f027916c4d61af007de8145c955c
I20260924 10:44:55.750650 9655 leader_election.cc:302] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f, b6aecda326bd41458ca1e6eae80da49b; no voters: 4c41f027916c4d61af007de8145c955c
I20260924 10:44:55.750690 10198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:44:55.750751 10198 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c41f027916c4d61af007de8145c955c)
I20260924 10:44:55.750840 10198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:44:55.751863 10198 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:55.752239 10198 leader_election.cc:288] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 election: Requested vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), b6aecda326bd41458ca1e6eae80da49b (127.2.182.60:45581)
I20260924 10:44:55.752380 10197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:44:55.752446 10197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c41f027916c4d61af007de8145c955c)
I20260924 10:44:55.752512 10197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:44:55.752592 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.752340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52968)
0924 10:44:55.752469 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.752526 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:44:55.752712 9681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b6aecda326bd41458ca1e6eae80da49b"
I20260924 10:44:55.753216 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.753033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52978)
0924 10:44:55.753129 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.753171 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:44:55.753374 9476 leader_election.cc:334] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.753767 10197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:55.753997 9681 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5acbcf659a64f5ea7e8858d307bb03f in current term 2: Already voted for candidate b6aecda326bd41458ca1e6eae80da49b in this term.
I20260924 10:44:55.754148 10197 leader_election.cc:288] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 election: Requested vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61:39393)
I20260924 10:44:55.754439 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6aecda326bd41458ca1e6eae80da49b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
I20260924 10:44:55.754491 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.754246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52988)
0924 10:44:55.754351 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.754413 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 10:44:55.754592 9499 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b6aecda326bd41458ca1e6eae80da49b in current term 2: Already voted for candidate a5acbcf659a64f5ea7e8858d307bb03f in this term.
I20260924 10:44:55.754801 9476 leader_election.cc:302] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f; no voters: 4c41f027916c4d61af007de8145c955c, b6aecda326bd41458ca1e6eae80da49b
I20260924 10:44:55.754956 10198 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 10:44:55.755215 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:55.754925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:52994)
0924 10:44:55.755128 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:55.755170 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260924 10:44:55.755321 9658 leader_election.cc:334] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:55.755383 9658 leader_election.cc:302] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6aecda326bd41458ca1e6eae80da49b; no voters: 4c41f027916c4d61af007de8145c955c, a5acbcf659a64f5ea7e8858d307bb03f
I20260924 10:44:55.755517 10197 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 10:44:56.231829 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:56.231360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:36791)
0924 10:44:56.231646 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:56.231717 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260924 10:44:56.233418 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:56.232855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:39281)
0924 10:44:56.233244 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:56.233309 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":115,"threads_started":1}
I20260924 10:44:56.234146 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:56.233781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:59429)
0924 10:44:56.233998 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:56.234053 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260924 10:44:57.233121 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.232626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:48551)
0924 10:44:57.232937 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.233009 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260924 10:44:57.234436 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.233986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:43293)
0924 10:44:57.234283 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.234342 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":66,"threads_started":1}
I20260924 10:44:57.235124 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.234762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:40165)
0924 10:44:57.234971 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.235034 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260924 10:44:57.732564 10213 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:57.732739 10213 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:57.733147 10213 leader_election.cc:288] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61:39393)
I20260924 10:44:57.733580 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6aecda326bd41458ca1e6eae80da49b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" is_pre_election: true
I20260924 10:44:57.733752 9499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6aecda326bd41458ca1e6eae80da49b in term 2.
I20260924 10:44:57.733809 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.733302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53296)
0924 10:44:57.733621 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.733706 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":59,"threads_started":1}
I20260924 10:44:57.734082 9655 leader_election.cc:302] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f, b6aecda326bd41458ca1e6eae80da49b; no voters:
I20260924 10:44:57.734273 10213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 10:44:57.734329 10213 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:57.734376 10213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:44:57.734630 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.734427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53300)
0924 10:44:57.734525 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.734576 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 10:44:57.734736 9658 leader_election.cc:334] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:57.735571 10213 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:57.735939 10213 leader_election.cc:288] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 election: Requested vote from peers 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999), a5acbcf659a64f5ea7e8858d307bb03f (127.2.182.61:39393)
I20260924 10:44:57.736222 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6aecda326bd41458ca1e6eae80da49b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5acbcf659a64f5ea7e8858d307bb03f"
I20260924 10:44:57.736327 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.736019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53316)
0924 10:44:57.736197 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.736251 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 10:44:57.736387 9499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:44:57.737043 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.736785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53318)
0924 10:44:57.736942 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.736998 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:44:57.737142 9658 leader_election.cc:334] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
I20260924 10:44:57.737582 9499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6aecda326bd41458ca1e6eae80da49b in term 3.
I20260924 10:44:57.737828 9655 leader_election.cc:302] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5acbcf659a64f5ea7e8858d307bb03f, b6aecda326bd41458ca1e6eae80da49b; no voters: 4c41f027916c4d61af007de8145c955c
I20260924 10:44:57.737986 10213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:44:57.738183 10213 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [term 3 LEADER]: Becoming Leader. State: Replica: b6aecda326bd41458ca1e6eae80da49b, State: Running, Role: LEADER
I20260924 10:44:57.738430 10213 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } }
I20260924 10:44:57.739598 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aecda326bd41458ca1e6eae80da49b. Latest consensus state: current_term: 3 leader_uuid: "b6aecda326bd41458ca1e6eae80da49b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:57.739709 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:57.739967 10218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:57.742043 10218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:57.742394 10218 catalog_manager.cc:1259] Loaded cluster ID: c9e4592b017045d2a59c0acecdbdfe92
I20260924 10:44:57.742457 10218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:57.742975 10218 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:57.743391 10218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Loaded TSK: 0
I20260924 10:44:57.744055 10218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:57.815076 9671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:44:57.818312 9671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:44:57.821208 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.820944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53332)
0924 10:44:57.821076 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.821133 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:44:57.821256 9499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [term 3 FOLLOWER]: Refusing update from remote peer b6aecda326bd41458ca1e6eae80da49b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 10:44:57.821703 10216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 10:44:57.821835 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.821679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.0.0.1:53336)
0924 10:44:57.821749 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.821790 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 10:44:57.821947 9658 consensus_peers.cc:487] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b -> Peer 4c41f027916c4d61af007de8145c955c (127.2.182.62:32999): Couldn't send request to peer 4c41f027916c4d61af007de8145c955c. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:44:57.823299 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aecda326bd41458ca1e6eae80da49b. Latest consensus state: current_term: 3 leader_uuid: "b6aecda326bd41458ca1e6eae80da49b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:57.823426 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:57.824501 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b6aecda326bd41458ca1e6eae80da49b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:57.824617 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:57.825044 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b6aecda326bd41458ca1e6eae80da49b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c41f027916c4d61af007de8145c955c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32999 } } peers { permanent_uuid: "a5acbcf659a64f5ea7e8858d307bb03f" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 39393 } attrs { promote: false } } peers { permanent_uuid: "b6aecda326bd41458ca1e6eae80da49b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45581 } attrs { promote: false } } }
I20260924 10:44:57.825162 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5acbcf659a64f5ea7e8858d307bb03f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:57.834904 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.830353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43595 (local address 127.0.0.1:48448)
0924 10:44:57.830560 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.830574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:57.830656 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:57.831770 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:57.831774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:57.831793 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:57.831874 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.831879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.832799 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.832805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.833736 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.833747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.834445 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.834448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.834602 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.834630 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:44:57.834716 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:44:57.834784 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":22}
I20260924 10:44:57.834945 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.831163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.2.182.1:43595)
0924 10:44:57.831560 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:57.831567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:57.831582 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:57.831629 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:57.831631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:57.831673 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:57.831754 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:57.831975 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.832688 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.833847 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.834323 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.834450 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.834529 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:57.834733 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:44:57.834814 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":60,"threads_started":1}
I20260924 10:44:57.835510 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.830926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:38709 (local address 127.0.0.1:55900)
0924 10:44:57.831435 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.831446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:57.831552 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:57.832038 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:57.832041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:57.832055 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:57.832127 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.832132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.833005 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.833008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.834102 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.834112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.835085 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.835089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.835202 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.835224 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:44:57.835298 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:44:57.835367 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":70,"threads_started":1}
I20260924 10:44:57.835630 9784 tablet_service.cc:1467] Processing CreateTablet for tablet c43c68e3ea724428a4553abe3332c114 (DEFAULT_TABLE table=test-table [id=69c945b858ee46f1a665cf1876925f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:57.835969 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c68e3ea724428a4553abe3332c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:57.835999 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.830640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:38355 (local address 127.0.0.1:60638)
0924 10:44:57.830851 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.830863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:57.831343 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:57.831602 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:57.831607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:57.831623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:44:57.831745 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.831752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.832824 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.832829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.834625 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:57.834634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.835627 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.835630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:57.835723 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.835746 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:44:57.835820 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:44:57.835886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":37,"thread_start_us":85,"threads_started":1}
I20260924 10:44:57.836092 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.830985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.2.182.2:38355)
0924 10:44:57.831252 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:57.831258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:57.831301 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:44:57.831426 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:57.831429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:57.831477 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:57.831592 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:57.831861 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.832675 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.834757 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.835382 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.835472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.835549 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:57.835876 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:44:57.835962 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 10:44:57.836668 9921 tablet_service.cc:1467] Processing CreateTablet for tablet c43c68e3ea724428a4553abe3332c114 (DEFAULT_TABLE table=test-table [id=69c945b858ee46f1a665cf1876925f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:57.836977 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c68e3ea724428a4553abe3332c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:57.837105 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.831033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.2.182.3:38709)
0924 10:44:57.831303 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:57.831307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:57.831520 (+ 213us) server_negotiation.cc:407] Connection header received
0924 10:44:57.831896 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:57.831902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:57.831962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:57.832037 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:44:57.832228 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.832723 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.834233 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:57.834873 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:57.834964 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:57.835031 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:57.835336 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:44:57.836964 (+ 1628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260924 10:44:57.838337 10231 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: Bootstrap starting.
I20260924 10:44:57.839269 10232 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Bootstrap starting.
I20260924 10:44:57.839468 10231 tablet_bootstrap.cc:654] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:57.839767 10057 tablet_service.cc:1467] Processing CreateTablet for tablet c43c68e3ea724428a4553abe3332c114 (DEFAULT_TABLE table=test-table [id=69c945b858ee46f1a665cf1876925f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:44:57.840029 10057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c68e3ea724428a4553abe3332c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:57.840152 10232 tablet_bootstrap.cc:654] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:57.841241 10232 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: No bootstrap required, opened a new log
I20260924 10:44:57.841347 10232 ts_tablet_manager.cc:1397] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:57.841742 10232 raft_consensus.cc:348] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.841871 10232 raft_consensus.cc:374] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:57.841933 10232 raft_consensus.cc:729] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54411e80dac479ba742837550cdedcc, State: Initialized, Role: FOLLOWER
I20260924 10:44:57.842093 10232 consensus_queue.cc:260] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.842504 10232 ts_tablet_manager.cc:1428] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:44:57.842967 10231 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: No bootstrap required, opened a new log
I20260924 10:44:57.843104 10231 ts_tablet_manager.cc:1397] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 10:44:57.843494 10231 raft_consensus.cc:348] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.843590 10231 raft_consensus.cc:374] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:57.843617 10231 raft_consensus.cc:729] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ace47fb8494fe7addfe54733d314dc, State: Initialized, Role: FOLLOWER
I20260924 10:44:57.843603 10235 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Bootstrap starting.
I20260924 10:44:57.843742 10231 consensus_queue.cc:260] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.844064 10231 ts_tablet_manager.cc:1428] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:57.844830 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.844443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:41107)
0924 10:44:57.844687 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.844746 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260924 10:44:57.845281 10235 tablet_bootstrap.cc:654] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:57.845973 9987 heartbeater.cc:503] Master 127.2.182.60:45581 was elected leader, sending a full tablet report...
I20260924 10:44:57.846174 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.843171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:57899)
0924 10:44:57.846030 (+ 2859us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.846083 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2803,"thread_start_us":90,"threads_started":1}
I20260924 10:44:57.846952 10235 tablet_bootstrap.cc:492] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: No bootstrap required, opened a new log
I20260924 10:44:57.847012 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.846630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:55367)
0924 10:44:57.846892 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.846944 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260924 10:44:57.847079 10235 ts_tablet_manager.cc:1397] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:44:57.847380 9850 heartbeater.cc:503] Master 127.2.182.60:45581 was elected leader, sending a full tablet report...
I20260924 10:44:57.847635 10235 raft_consensus.cc:348] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.847764 10235 raft_consensus.cc:374] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:57.847827 10235 raft_consensus.cc:729] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04be2753ef0b4fd79f23ab480a3c141a, State: Initialized, Role: FOLLOWER
I20260924 10:44:57.847918 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.847585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:42917)
0924 10:44:57.847810 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.847859 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260924 10:44:57.847958 10235 consensus_queue.cc:260] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:57.848248 10235 ts_tablet_manager.cc:1428] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:44:57.849522 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.848701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:46845)
0924 10:44:57.849360 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.849437 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":85,"threads_started":1}
I20260924 10:44:57.849891 10123 heartbeater.cc:503] Master 127.2.182.60:45581 was elected leader, sending a full tablet report...
I20260924 10:44:57.850171 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:57.849967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:54647)
0924 10:44:57.850062 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:57.850110 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 10:44:58.010195 9853 tablet.cc:2369] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:58.049594 10238 raft_consensus.cc:482] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:44:58.049738 10238 raft_consensus.cc:504] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:58.050173 10238 leader_election.cc:288] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a54411e80dac479ba742837550cdedcc (127.2.182.2:38355), 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:58.050650 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c68e3ea724428a4553abe3332c114" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54411e80dac479ba742837550cdedcc" is_pre_election: true
I20260924 10:44:58.050668 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c68e3ea724428a4553abe3332c114" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04be2753ef0b4fd79f23ab480a3c141a" is_pre_election: true
I20260924 10:44:58.050782 9941 raft_consensus.cc:2454] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 0.
I20260924 10:44:58.050840 10077 raft_consensus.cc:2454] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 0.
I20260924 10:44:58.051186 9739 leader_election.cc:302] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a54411e80dac479ba742837550cdedcc, e3ace47fb8494fe7addfe54733d314dc; no voters:
I20260924 10:44:58.051359 10238 raft_consensus.cc:2781] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:44:58.051417 10238 raft_consensus.cc:482] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:44:58.051445 10238 raft_consensus.cc:3037] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:58.053560 10238 raft_consensus.cc:504] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:58.053916 10238 leader_election.cc:288] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 election: Requested vote from peers a54411e80dac479ba742837550cdedcc (127.2.182.2:38355), 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709)
I20260924 10:44:58.054291 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c68e3ea724428a4553abe3332c114" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04be2753ef0b4fd79f23ab480a3c141a"
I20260924 10:44:58.054291 9941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c68e3ea724428a4553abe3332c114" candidate_uuid: "e3ace47fb8494fe7addfe54733d314dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54411e80dac479ba742837550cdedcc"
I20260924 10:44:58.054412 9941 raft_consensus.cc:3037] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:58.054412 10077 raft_consensus.cc:3037] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:58.055544 10077 raft_consensus.cc:2454] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 1.
I20260924 10:44:58.055544 9941 raft_consensus.cc:2454] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ace47fb8494fe7addfe54733d314dc in term 1.
I20260924 10:44:58.055876 9739 leader_election.cc:302] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a54411e80dac479ba742837550cdedcc, e3ace47fb8494fe7addfe54733d314dc; no voters:
I20260924 10:44:58.056041 10238 raft_consensus.cc:2781] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:58.056304 10238 raft_consensus.cc:686] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 1 LEADER]: Becoming Leader. State: Replica: e3ace47fb8494fe7addfe54733d314dc, State: Running, Role: LEADER
I20260924 10:44:58.056665 10238 consensus_queue.cc:237] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } }
I20260924 10:44:58.057065 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.056819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:37867)
0924 10:44:58.056933 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.057001 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:44:58.058833 9670 catalog_manager.cc:5495] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ace47fb8494fe7addfe54733d314dc (127.2.182.1). New cstate: current_term: 1 leader_uuid: "e3ace47fb8494fe7addfe54733d314dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ace47fb8494fe7addfe54733d314dc" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 } health_report { overall_health: UNKNOWN } } }
I20260924 10:44:58.125586 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.122964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.2.182.1:43595)
0924 10:44:58.123096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:58.123100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:58.123112 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:58.123156 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:58.123158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:58.123201 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:58.123267 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:58.123488 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:58.123960 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:58.124696 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:58.124920 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:58.124991 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:58.125179 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:58.125239 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:58.125287 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:58.125421 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:44:58.125484 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:44:58.127734 10077 raft_consensus.cc:1261] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Refusing update from remote peer e3ace47fb8494fe7addfe54733d314dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:58.127766 9941 raft_consensus.cc:1261] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Refusing update from remote peer e3ace47fb8494fe7addfe54733d314dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:58.128223 10244 consensus_queue.cc:1035] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04be2753ef0b4fd79f23ab480a3c141a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:58.128371 10238 consensus_queue.cc:1035] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a54411e80dac479ba742837550cdedcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:58.129830 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.129596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:43115)
0924 10:44:58.129681 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.129741 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:44:58.130455 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.130229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:39267)
0924 10:44:58.130329 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.130389 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:44:58.133490 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.133207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.2:52897)
0924 10:44:58.133376 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.133429 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 10:44:58.135459 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.135170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:48033)
0924 10:44:58.135331 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.135385 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:44:58.137269 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.136863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.3:43355)
0924 10:44:58.137129 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.137183 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260924 10:44:58.138453 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.138203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32999 (local address 127.2.182.1:36105)
0924 10:44:58.138339 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.138391 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:44:58.139907 9671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:58.140076 9671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:44:58.143544 9671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6aecda326bd41458ca1e6eae80da49b: Sending DeleteTablet for 3 replicas of tablet c43c68e3ea724428a4553abe3332c114
I20260924 10:44:58.144282 10057 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c68e3ea724428a4553abe3332c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:58 UTC) from {username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:55900
I20260924 10:44:58.144299 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c68e3ea724428a4553abe3332c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:58 UTC) from {username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:60638
I20260924 10:44:58.144404 9784 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c68e3ea724428a4553abe3332c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:44:58 UTC) from {username='oryx', principal='oryx/127.2.182.60@KRBTEST.COM'} at 127.0.0.1:48448
I20260924 10:44:58.144655 10256 tablet_replica.cc:330] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: stopping tablet replica
I20260924 10:44:58.144660 10257 tablet_replica.cc:330] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: stopping tablet replica
I20260924 10:44:58.144709 10258 tablet_replica.cc:330] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc: stopping tablet replica
I20260924 10:44:58.144795 10256 raft_consensus.cc:2229] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:58.144796 10257 raft_consensus.cc:2229] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:44:58.144850 10258 raft_consensus.cc:2229] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:44:58.144959 10257 raft_consensus.cc:2258] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:58.144964 10256 raft_consensus.cc:2258] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:58.145088 10258 raft_consensus.cc:2258] T c43c68e3ea724428a4553abe3332c114 P e3ace47fb8494fe7addfe54733d314dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:44:58.145352 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9633
I20260924 10:44:58.145802 10257 ts_tablet_manager.cc:1905] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:58.146111 10256 ts_tablet_manager.cc:1905] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:44:58.148734 10256 ts_tablet_manager.cc:1918] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:58.148830 10256 log.cc:1196] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/c43c68e3ea724428a4553abe3332c114
I20260924 10:44:58.149091 10257 ts_tablet_manager.cc:1918] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:44:58.149211 10257 log.cc:1196] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/c43c68e3ea724428a4553abe3332c114
I20260924 10:44:58.149219 10256 ts_tablet_manager.cc:1939] T c43c68e3ea724428a4553abe3332c114 P 04be2753ef0b4fd79f23ab480a3c141a: Deleting consensus metadata
I20260924 10:44:58.149592 10257 ts_tablet_manager.cc:1939] T c43c68e3ea724428a4553abe3332c114 P a54411e80dac479ba742837550cdedcc: Deleting consensus metadata
I20260924 10:44:58.149855 9658 catalog_manager.cc:4841] TS 04be2753ef0b4fd79f23ab480a3c141a (127.2.182.3:38709): tablet c43c68e3ea724428a4553abe3332c114 (table test-table [id=69c945b858ee46f1a665cf1876925f3c]) successfully deleted
I20260924 10:44:58.150302 9657 catalog_manager.cc:4841] TS a54411e80dac479ba742837550cdedcc (127.2.182.2:38355): tablet c43c68e3ea724428a4553abe3332c114 (table test-table [id=69c945b858ee46f1a665cf1876925f3c]) successfully deleted
I20260924 10:44:58.154024 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9858
I20260924 10:44:58.154182 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:58.153889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:43595 (local address 127.0.0.1:48470)
0924 10:44:58.154014 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:58.154089 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:43595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:44:58.154995 9656 catalog_manager.cc:4584] TS e3ace47fb8494fe7addfe54733d314dc (127.2.182.1:43595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c43c68e3ea724428a4553abe3332c114: Network error: Client connection negotiation failed: client connection to 127.2.182.1:43595: connect: Connection refused (error 111)
I20260924 10:44:58.159817 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9994
I20260924 10:44:58.165345 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9351
I20260924 10:44:58.170785 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 9453
2026-09-24T10:44:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7535 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:58 dist-test-slave-6lwt krb5kdc[10261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:58 dist-test-slave-6lwt krb5kdc[10261](info): set up 2 sockets
Sep 24 10:44:58 dist-test-slave-6lwt krb5kdc[10261](info): commencing operation
krb5kdc: starting...
W20260924 10:44:59.197587 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab.
Entry for principal oryx/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab.
I20260924 10:44:59.315415 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46211
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab
--principal=oryx/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:59.418249 10274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:59.418475 10274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:59.418512 10274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:59.422317 10274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:59.422361 10274 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:44:59.422384 10274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:59.422401 10274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:59.422416 10274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:59.422431 10274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:59.427294 10274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.62.keytab
--principal=oryx/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:59.428092 10274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:59.428987 10274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:59.436728 10279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:59.436793 10280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:59.436743 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:59.436790 10274 server_base.cc:1034] running on GCE node
I20260924 10:44:59.437459 10274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:59.438448 10274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:59.439643 10274 hybrid_clock.cc:648] HybridClock initialized: now 1790246699439634 us; error 26 us; skew 500 ppm
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:59.442206 10274 init.cc:377] Logged in from keytab as oryx/127.2.182.62@KRBTEST.COM (short username oryx)
I20260924 10:44:59.442962 10274 webserver.cc:460] Webserver started at http://127.2.182.62:41717/ using document root <none> and password file <none>
I20260924 10:44:59.443536 10274 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:59.443640 10274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:59.443859 10274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:59.445745 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "277f043b424347f992f36cbc3c6cea82"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "22406f3b3e2ea10dabe2228ee199335c"
server_key_iv: "3716b9c299aa6e22fba6d8b60bb79a14"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.446307 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "277f043b424347f992f36cbc3c6cea82"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "22406f3b3e2ea10dabe2228ee199335c"
server_key_iv: "3716b9c299aa6e22fba6d8b60bb79a14"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.449757 10274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:59.451972 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:59.453116 10274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:59.453239 10274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "277f043b424347f992f36cbc3c6cea82"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "22406f3b3e2ea10dabe2228ee199335c"
server_key_iv: "3716b9c299aa6e22fba6d8b60bb79a14"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.453339 10274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:59.471153 10274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:59.473958 10274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:59.474157 10274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:59.481611 10274 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:46211
I20260924 10:44:59.481734 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:46211 every 8 connection(s)
I20260924 10:44:59.482533 10274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:59.485280 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:59.490492 10352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Bootstrap starting.
I20260924 10:44:59.490934 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 10274
I20260924 10:44:59.491109 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:44:59.491374 2776 external_mini_cluster.cc:1428] Setting key 086a451114048b2781c808a4cbb31976
I20260924 10:44:59.493119 10352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:59.493860 10352 log.cc:824] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:59.495673 10352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: No bootstrap required, opened a new log
I20260924 10:44:59.499737 10352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } }
I20260924 10:44:59.500033 10352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:59.500094 10352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277f043b424347f992f36cbc3c6cea82, State: Initialized, Role: FOLLOWER
I20260924 10:44:59.500584 10352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } }
I20260924 10:44:59.500742 10352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:59.500826 10352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:59.500938 10352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:59.501752 10352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } }
I20260924 10:44:59.502072 10352 leader_election.cc:302] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 277f043b424347f992f36cbc3c6cea82; no voters:
I20260924 10:44:59.502410 10352 leader_election.cc:288] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:59.502451 10357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:59.502635 10357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [term 1 LEADER]: Becoming Leader. State: Replica: 277f043b424347f992f36cbc3c6cea82, State: Running, Role: LEADER
I20260924 10:44:59.502982 10357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } }
I20260924 10:44:59.503811 10352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:59.504221 10359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "277f043b424347f992f36cbc3c6cea82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } } }
I20260924 10:44:59.504421 10359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:59.504671 10358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277f043b424347f992f36cbc3c6cea82. Latest consensus state: current_term: 1 leader_uuid: "277f043b424347f992f36cbc3c6cea82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f043b424347f992f36cbc3c6cea82" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46211 } } }
I20260924 10:44:59.504783 10358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:59.505009 10362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:59.506661 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.492582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.2.182.62:46211)
0924 10:44:59.492996 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:59.493011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:59.493045 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:59.493687 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:59.493705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:59.494060 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:59.494394 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:59.495195 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.496222 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.497187 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.497916 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.498082 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.500684 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:59.500689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:59.500701 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:59.500748 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:59.502716 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.503386 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.503391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.503395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.503510 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.503823 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.503826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.503828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.504385 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:59.504616 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:59.505384 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 10:44:59.505699 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260924 10:44:59.508332 10362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:59.514109 10362 catalog_manager.cc:1347] Generated new cluster ID: c9eaf7ec07dd4dd09e063e5527209c47
I20260924 10:44:59.514179 10362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:59.522552 10362 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:59.523721 10362 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:59.528744 10362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Generated new TSK 0
I20260924 10:44:59.529371 10362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
I20260924 10:44:59.558368 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46211
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:59.665309 10378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:59.665542 10378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:59.665583 10378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:59.669471 10378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:59.669521 10378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:59.669576 10378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:59.669607 10378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:44:59.674367 10378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:59.675158 10378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:59.676034 10378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:59.685030 10383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:59.685019 10386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:59.685025 10384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:59.685302 10378 server_base.cc:1034] running on GCE node
I20260924 10:44:59.685784 10378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:59.686343 10378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:59.687526 10378 hybrid_clock.cc:648] HybridClock initialized: now 1790246699687512 us; error 31 us; skew 500 ppm
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:59.690172 10378 init.cc:377] Logged in from keytab as oryx/127.2.182.1@KRBTEST.COM (short username oryx)
I20260924 10:44:59.690946 10378 webserver.cc:460] Webserver started at http://127.2.182.1:39907/ using document root <none> and password file <none>
I20260924 10:44:59.691452 10378 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:59.691535 10378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:59.691746 10378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:59.693682 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "6e19a16896c96a74df943591f4ee0165"
server_key_iv: "33413aedc3e30599861d9f44419ae4e1"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.694203 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "6e19a16896c96a74df943591f4ee0165"
server_key_iv: "33413aedc3e30599861d9f44419ae4e1"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.697623 10378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:59.699865 10393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:59.701067 10378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:59.701195 10378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "6e19a16896c96a74df943591f4ee0165"
server_key_iv: "33413aedc3e30599861d9f44419ae4e1"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.701301 10378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:59.717620 10378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:59.720287 10378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:59.720500 10378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:59.721915 10378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:59.721990 10378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:59.722062 10378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:59.722107 10378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:59.733265 10378 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40613
I20260924 10:44:59.733330 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40613 every 8 connection(s)
I20260924 10:44:59.734110 10378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:44:59.735623 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.722780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:48513 (local address 127.2.182.62:46211)
0924 10:44:59.722902 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:59.722906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:59.723973 (+ 1067us) server_negotiation.cc:407] Connection header received
0924 10:44:59.725095 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:59.725098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:59.725147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:59.725193 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:59.726764 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.727285 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.728610 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.729022 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.729139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.731900 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:59.731908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:59.731910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:59.731932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:59.733587 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.734147 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.734152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.734154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.734236 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.734612 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.734615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.734616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.734828 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:59.734924 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:59.735402 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:44:59.735506 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:44:59.736418 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.723324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.1:48513)
0924 10:44:59.723837 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:59.723879 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:59.724801 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:59.725567 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:59.725575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:59.726025 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:59.726604 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.726621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.727413 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.727419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.728474 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.728484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.729184 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.729188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.729364 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.729959 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:59.729986 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:59.731751 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:59.733692 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.733698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.733711 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.734048 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.734335 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.734338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.734340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.734474 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.734922 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:59.734927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:59.735196 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 10:44:59.735356 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 10:44:59.735596 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260924 10:44:59.737565 10509 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46211
I20260924 10:44:59.737694 10509 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:59.738222 10509 heartbeater.cc:511] Master 127.2.182.62:46211 requested a full tablet report, sending...
I20260924 10:44:59.739778 10305 ts_manager.cc:194] Registered new tserver with Master: 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613)
I20260924 10:44:59.740986 10305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:48513
I20260924 10:44:59.744256 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 10378
I20260924 10:44:59.744392 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:44:59.744745 2776 external_mini_cluster.cc:1428] Setting key 44338b42bce3405ef5be1fbbdec42b4f
I20260924 10:44:59.746944 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.739724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.1:51407)
0924 10:44:59.740045 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:59.740057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:59.740162 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:59.740405 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:59.740408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:59.740589 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:59.740678 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.740683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.741621 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.741625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.742274 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.742284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.742860 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.742864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.742949 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.743395 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:59.743414 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:59.743734 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:59.745514 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.745517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.745519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.745766 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.746023 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.746026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.746028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.746126 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.746552 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:59.746554 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:59.746625 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:44:59.746709 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:44:59.746804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
I20260924 10:44:59.747120 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.739796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:51407 (local address 127.2.182.62:46211)
0924 10:44:59.739910 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:59.739914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:59.740125 (+ 211us) server_negotiation.cc:407] Connection header received
0924 10:44:59.740260 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:59.740263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:59.740304 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:59.740391 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:59.740787 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.741443 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.742430 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.742741 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.742847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.743852 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:59.743855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:59.743857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:59.743877 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:59.745407 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.745865 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.745867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.745868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.745935 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.746222 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.746225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.746226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.746442 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:59.746555 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:59.746784 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:44:59.746905 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for oryx/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
I20260924 10:44:59.771189 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46211
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:59.875453 10518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:59.875679 10518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:59.875717 10518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:59.879604 10518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:59.879657 10518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:59.879694 10518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:44:59.879725 10518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:44:59.884388 10518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:59.885308 10518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:59.886284 10518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:59.895758 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:59.895932 10518 server_base.cc:1034] running on GCE node
W20260924 10:44:59.895789 10526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:59.895764 10523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:59.896457 10518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:59.897056 10518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:59.898240 10518 hybrid_clock.cc:648] HybridClock initialized: now 1790246699898229 us; error 24 us; skew 500 ppm
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:59.900714 10518 init.cc:377] Logged in from keytab as oryx/127.2.182.2@KRBTEST.COM (short username oryx)
I20260924 10:44:59.901563 10518 webserver.cc:460] Webserver started at http://127.2.182.2:35653/ using document root <none> and password file <none>
I20260924 10:44:59.902083 10518 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:59.902166 10518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:59.902376 10518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:59.904553 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "b79e1706683247f89155284b0431393c"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "83537bd7515f28d9a000a68cdeca91a8"
server_key_iv: "889f1b24b0a5f077014df248171050c2"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.905126 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "b79e1706683247f89155284b0431393c"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "83537bd7515f28d9a000a68cdeca91a8"
server_key_iv: "889f1b24b0a5f077014df248171050c2"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.908600 10518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:44:59.910822 10533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:59.911895 10518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:59.912038 10518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "b79e1706683247f89155284b0431393c"
format_stamp: "Formatted at 2026-09-24 10:44:59 on dist-test-slave-6lwt"
server_key: "83537bd7515f28d9a000a68cdeca91a8"
server_key_iv: "889f1b24b0a5f077014df248171050c2"
server_key_version: "encryptionkey@0"
I20260924 10:44:59.912135 10518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:59.940332 10518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:59.943041 10518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:59.943279 10518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:59.945161 10518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:44:59.945257 10518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:59.945339 10518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:44:59.945386 10518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:44:59 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:44:59.956485 10518 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34165
I20260924 10:44:59.956815 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34165 every 8 connection(s)
I20260924 10:44:59.957474 10518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:44:59.960039 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.946092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:51895 (local address 127.2.182.62:46211)
0924 10:44:59.946283 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:59.946288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:59.947514 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 10:44:59.948629 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:59.948632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:59.948690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:59.948793 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:59.950307 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.950929 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.952724 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.953075 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.953199 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.956138 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:59.956142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:59.956144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:59.956171 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:59.957971 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.958508 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.958531 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.958535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.958625 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.958970 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.958974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.958976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.959159 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:59.959264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:59.959735 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 10:44:59.959859 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:44:59.960718 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.946911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.2:51895)
0924 10:44:59.947363 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:59.947402 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:59.948393 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:59.948999 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:59.949008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:59.949460 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:59.950121 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.950140 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.951061 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.951064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.952598 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.952608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.953242 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.953246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.953424 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.954544 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:59.954560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:59.955934 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:59.958095 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.958100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.958113 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.958420 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.958727 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.958731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.958734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.958874 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.959272 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:59.959281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:59.959510 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 10:44:59.959689 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 10:44:59.959989 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260924 10:44:59.961843 10649 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46211
I20260924 10:44:59.961999 10649 heartbeater.cc:464] Registering TS with master...
I20260924 10:44:59.962500 10649 heartbeater.cc:511] Master 127.2.182.62:46211 requested a full tablet report, sending...
I20260924 10:44:59.963522 10306 ts_manager.cc:194] Registered new tserver with Master: b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:44:59.964298 10306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:51895
I20260924 10:44:59.967185 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 10518
I20260924 10:44:59.967336 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:44:59.967716 2776 external_mini_cluster.cc:1428] Setting key a97951fd7b7502f38a2a8ca6f4e0bb82
I20260924 10:44:59.971211 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.964115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:45319 (local address 127.2.182.62:46211)
0924 10:44:59.964220 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:59.964223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:59.964406 (+ 183us) server_negotiation.cc:407] Connection header received
0924 10:44:59.964512 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:59.964515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:59.964580 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:59.964654 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:59.965135 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.965871 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.966608 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.966805 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.966881 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.968110 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:59.968113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:59.968116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:59.968142 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:59.969584 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.970047 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.970050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.970051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.970118 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:59.970390 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:59.970391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:59.970392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:59.970571 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:59.970703 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:59.970984 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 10:44:59.971086 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:44:59.971222 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:59.964040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.2:45319)
0924 10:44:59.964323 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:59.964338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:59.964423 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:59.964687 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:59.964689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:59.964877 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:59.965030 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.965035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.965990 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.965992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.966511 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:59.966517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:59.966897 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:59.966900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:59.967022 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:59.967562 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:59.967578 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:59.967967 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:59.969692 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.969699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.969704 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.969959 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.970201 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:59.970206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:59.970211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:59.970307 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:59.970694 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:59.970698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:59.970792 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 10:44:59.970927 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:44:59.971061 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
I20260924 10:44:59.993321 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46211
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:00.097597 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:00.097883 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:00.098021 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:00.102712 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:00.102785 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:00.102877 10658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:45:00.102949 10658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:00.108445 10658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.62:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:00.109458 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:00.110388 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:00.119726 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:00.119755 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:00.119855 10658 server_base.cc:1034] running on GCE node
W20260924 10:45:00.119756 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:00.120291 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:00.120863 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:00.122076 10658 hybrid_clock.cc:648] HybridClock initialized: now 1790246700122065 us; error 24 us; skew 500 ppm
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:00.124756 10658 init.cc:377] Logged in from keytab as oryx/127.2.182.3@KRBTEST.COM (short username oryx)
I20260924 10:45:00.125625 10658 webserver.cc:460] Webserver started at http://127.2.182.3:34161/ using document root <none> and password file <none>
I20260924 10:45:00.126169 10658 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:00.126258 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:00.126495 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:00.128438 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "548d1c5a59964182be8470aa5f8a11d8"
format_stamp: "Formatted at 2026-09-24 10:45:00 on dist-test-slave-6lwt"
server_key: "22e050e5e5a09af9f5db38b8a1a5edf4"
server_key_iv: "541831d39197371ee8ae14470b4333bc"
server_key_version: "encryptionkey@0"
I20260924 10:45:00.129031 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "548d1c5a59964182be8470aa5f8a11d8"
format_stamp: "Formatted at 2026-09-24 10:45:00 on dist-test-slave-6lwt"
server_key: "22e050e5e5a09af9f5db38b8a1a5edf4"
server_key_iv: "541831d39197371ee8ae14470b4333bc"
server_key_version: "encryptionkey@0"
I20260924 10:45:00.132426 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:45:00.134589 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:00.135674 10658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:00.135795 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "548d1c5a59964182be8470aa5f8a11d8"
format_stamp: "Formatted at 2026-09-24 10:45:00 on dist-test-slave-6lwt"
server_key: "22e050e5e5a09af9f5db38b8a1a5edf4"
server_key_iv: "541831d39197371ee8ae14470b4333bc"
server_key_version: "encryptionkey@0"
I20260924 10:45:00.135916 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:00.150568 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:00.153730 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:00.153923 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:00.155215 10658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:00.155296 10658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:00.155411 10658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:00.155454 10658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:45:00.166353 10658 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:34763
I20260924 10:45:00.166436 10788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:34763 every 8 connection(s)
I20260924 10:45:00.167433 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:45:00.168596 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 10658
I20260924 10:45:00.168704 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:45:00.169034 2776 external_mini_cluster.cc:1428] Setting key 08ca7acfcf8ab0d3dff112928b8fc7de
I20260924 10:45:00.169919 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.156160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:55201 (local address 127.2.182.62:46211)
0924 10:45:00.156287 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.156291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.157431 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 10:45:00.158410 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.158414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.158475 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.158555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.160287 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.161211 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.162892 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.163301 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.163411 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.166190 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:00.166194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:00.166196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:00.166235 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:00.167737 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.168318 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.168322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.168324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.168402 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.168757 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.168761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.168762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.169010 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:00.169113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.169637 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.169749 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:45:00.170704 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.156714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.3:55201)
0924 10:45:00.157253 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.157333 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.158169 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.159018 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.159028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.159473 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:00.160103 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.160126 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.161358 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.161361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.162785 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.162796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.163443 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.163446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.163630 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.164226 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:00.164247 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:00.166013 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:00.167848 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.167854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.167867 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.168210 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.168504 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.168507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.168509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.168638 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.169110 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:00.169117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:00.169374 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.169581 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.169872 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":151,"threads_started":1}
I20260924 10:45:00.171857 10789 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46211
I20260924 10:45:00.172012 10789 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:00.172521 10789 heartbeater.cc:511] Master 127.2.182.62:46211 requested a full tablet report, sending...
I20260924 10:45:00.173519 10306 ts_manager.cc:194] Registered new tserver with Master: 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763)
I20260924 10:45:00.174187 10306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:55201
I20260924 10:45:00.180226 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.173826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:33209 (local address 127.2.182.62:46211)
0924 10:45:00.173932 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.173936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.174093 (+ 157us) server_negotiation.cc:407] Connection header received
0924 10:45:00.174197 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.174200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.174253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.174341 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.174735 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.175228 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.176050 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.176279 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.176385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.177384 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:00.177387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:00.177390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:00.177411 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:00.178679 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.179083 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.179087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.179089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.179171 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.179443 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.179448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.179450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.179617 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:00.179724 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.179975 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.180085 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:45:00.180235 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.173777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.2.182.3:33209)
0924 10:45:00.174019 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.174030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.174128 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.174353 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.174356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.174550 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:00.174633 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.174638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.175344 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.175347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.175928 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.175935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.176387 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.176390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.176485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.176936 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:00.176950 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:00.177258 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:00.178786 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.178792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.178795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.178985 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.179261 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.179265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.179267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.179353 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.179713 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:00.179716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:00.179811 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.179914 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.180047 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260924 10:45:00.182796 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:00.205060 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:00.198751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.2.182.62:46211)
0924 10:45:00.198890 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.198893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.198921 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:45:00.198959 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.198961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.199003 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.199096 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.199862 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.200368 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.201264 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.201474 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.201583 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.204824 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39668: BlockingRecv error: recv got EOF from 127.0.0.1:39668 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:45:00.213837 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.206425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.2.182.62:46211)
0924 10:45:00.206543 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.206547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.206654 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:45:00.206737 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.206739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.206775 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.206837 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.207402 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.207871 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.208710 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.208934 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.209022 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.211168 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:00.211171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:00.211173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:00.211193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:00.212347 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.212722 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.212725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.212726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.212795 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.213083 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.213085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.213086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.213246 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:00.213347 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.213542 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.213628 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:45:00.216823 10306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:45:00.219789 10306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:00.233878 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.228675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.2.182.2:34165)
0924 10:45:00.229046 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.229061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.229369 (+ 308us) server_negotiation.cc:407] Connection header received
0924 10:45:00.229448 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.229453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.229610 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.229722 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.230085 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.231160 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.232550 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233186 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.233484 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.233639 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.233738 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
I20260924 10:45:00.233880 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.227834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:34763 (local address 127.0.0.1:44470)
0924 10:45:00.228644 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.228657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.228750 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.229442 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.229447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.229465 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.229612 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.229622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.231029 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.231033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.232354 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.232360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233345 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.233438 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.233484 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.233568 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.233703 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":137,"thread_start_us":105,"threads_started":1}
I20260924 10:45:00.233870 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.228344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34165 (local address 127.0.0.1:56664)
0924 10:45:00.229187 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.229199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.229276 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.229802 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.229805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.229825 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.229953 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.229959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.231268 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.231271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.232428 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.232441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233311 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.233435 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.233482 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.233598 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.233714 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":130,"thread_start_us":104,"threads_started":1}
I20260924 10:45:00.234315 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.227679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40613 (local address 127.0.0.1:46590)
0924 10:45:00.228163 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.228209 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.228332 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.229641 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.229646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.229667 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.229752 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.229757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.230783 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.230789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.232341 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.232353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233957 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.234045 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.234065 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.234119 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.234187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":151,"thread_start_us":82,"threads_started":1}
I20260924 10:45:00.234423 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.228422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.2.182.3:34763)
0924 10:45:00.228712 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.228724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.228756 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:45:00.229056 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.229062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.229279 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.229454 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.229739 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.230911 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.232474 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233205 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233950 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.234132 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.234180 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.234267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":123,"threads_started":1}
I20260924 10:45:00.235769 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.227853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.2.182.1:40613)
0924 10:45:00.229127 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.229158 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.229182 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:45:00.229253 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.229259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.229471 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.229646 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.229858 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.230666 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.233040 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.233820 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.234613 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.235470 (+ 857us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.235530 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.235635 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":80,"threads_started":1}
I20260924 10:45:00.237349 10723 tablet_service.cc:1467] Processing CreateTablet for tablet eebb66e6ab354adbb1d9550cf239a355 (DEFAULT_TABLE table=test-table [id=88df88f84336494083cf5c4f297dfc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.237392 10583 tablet_service.cc:1467] Processing CreateTablet for tablet eebb66e6ab354adbb1d9550cf239a355 (DEFAULT_TABLE table=test-table [id=88df88f84336494083cf5c4f297dfc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.238376 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebb66e6ab354adbb1d9550cf239a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.238408 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebb66e6ab354adbb1d9550cf239a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.238168 10443 tablet_service.cc:1467] Processing CreateTablet for tablet eebb66e6ab354adbb1d9550cf239a355 (DEFAULT_TABLE table=test-table [id=88df88f84336494083cf5c4f297dfc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.239019 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebb66e6ab354adbb1d9550cf239a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.245180 10817 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Bootstrap starting.
I20260924 10:45:00.246299 10818 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Bootstrap starting.
I20260924 10:45:00.247213 10817 tablet_bootstrap.cc:654] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.247932 10817 log.cc:824] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:00.248437 10818 tablet_bootstrap.cc:654] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.248481 10819 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Bootstrap starting.
I20260924 10:45:00.249259 10818 log.cc:824] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:00.249935 10817 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: No bootstrap required, opened a new log
I20260924 10:45:00.250155 10817 ts_tablet_manager.cc:1397] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:45:00.250695 10819 tablet_bootstrap.cc:654] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.251266 10819 log.cc:824] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:00.251315 10818 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: No bootstrap required, opened a new log
I20260924 10:45:00.251470 10818 ts_tablet_manager.cc:1397] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:45:00.252732 10819 tablet_bootstrap.cc:492] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: No bootstrap required, opened a new log
I20260924 10:45:00.252948 10819 ts_tablet_manager.cc:1397] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:45:00.253927 10817 raft_consensus.cc:348] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.254184 10817 raft_consensus.cc:374] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.254262 10817 raft_consensus.cc:729] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.254719 10817 consensus_queue.cc:260] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.255057 10818 raft_consensus.cc:348] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.255314 10818 raft_consensus.cc:374] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.255334 10789 heartbeater.cc:503] Master 127.2.182.62:46211 was elected leader, sending a full tablet report...
I20260924 10:45:00.255431 10818 raft_consensus.cc:729] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff9f7a9567b49ffaae34148f21fcaf7, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.255679 10817 ts_tablet_manager.cc:1428] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:45:00.255910 10818 consensus_queue.cc:260] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.256610 10509 heartbeater.cc:503] Master 127.2.182.62:46211 was elected leader, sending a full tablet report...
I20260924 10:45:00.256837 10818 ts_tablet_manager.cc:1428] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:45:00.257337 10819 raft_consensus.cc:348] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.257591 10819 raft_consensus.cc:374] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.257653 10819 raft_consensus.cc:729] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e1706683247f89155284b0431393c, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.258047 10819 consensus_queue.cc:260] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.258663 10649 heartbeater.cc:503] Master 127.2.182.62:46211 was elected leader, sending a full tablet report...
I20260924 10:45:00.258924 10819 ts_tablet_manager.cc:1428] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:45:00.352662 10825 raft_consensus.cc:482] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:00.352929 10825 raft_consensus.cc:504] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.354085 10825 leader_election.cc:288] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763)
I20260924 10:45:00.358201 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.354370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:50733 (local address 127.2.182.1:40613)
0924 10:45:00.354505 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.354509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.354530 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:45:00.354718 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.354721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.354778 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.354865 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.355175 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.355789 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.357136 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.357643 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.357719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.357767 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.358004 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.358079 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:45:00.358207 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.354314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40613 (local address 127.2.182.2:50733)
0924 10:45:00.354472 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.354488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.354588 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.354913 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.354916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.354938 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.355060 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.355070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.355916 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.355919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.356984 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.356994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.357750 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.357753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.357871 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.357897 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.357977 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.358046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260924 10:45:00.358594 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.354420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33309 (local address 127.2.182.3:34763)
0924 10:45:00.354531 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.354535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.354910 (+ 375us) server_negotiation.cc:407] Connection header received
0924 10:45:00.354988 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.354991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.355044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.355123 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.355390 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.356035 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.357481 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.358013 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.358104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.358156 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.358399 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.358477 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:45:00.358924 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.354325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:34763 (local address 127.2.182.2:33309)
0924 10:45:00.354804 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.354819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.354910 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.355158 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.355160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.355176 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.355288 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.355293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.356139 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.356141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.357365 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.357374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.358127 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.358131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.358232 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.358259 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.358724 (+ 465us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.358789 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":105,"thread_start_us":145,"threads_started":1}
I20260924 10:45:00.358883 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebb66e6ab354adbb1d9550cf239a355" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" is_pre_election: true
I20260924 10:45:00.359192 10463 raft_consensus.cc:2454] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 0.
I20260924 10:45:00.359555 10743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebb66e6ab354adbb1d9550cf239a355" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548d1c5a59964182be8470aa5f8a11d8" is_pre_election: true
I20260924 10:45:00.359696 10535 leader_election.cc:302] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.359872 10743 raft_consensus.cc:2454] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 0.
I20260924 10:45:00.360018 10825 raft_consensus.cc:2781] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:00.360105 10825 raft_consensus.cc:482] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:00.360157 10825 raft_consensus.cc:3037] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.361301 10825 raft_consensus.cc:504] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.361685 10825 leader_election.cc:288] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 election: Requested vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763)
I20260924 10:45:00.361948 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebb66e6ab354adbb1d9550cf239a355" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
I20260924 10:45:00.362059 10743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebb66e6ab354adbb1d9550cf239a355" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548d1c5a59964182be8470aa5f8a11d8"
I20260924 10:45:00.362064 10463 raft_consensus.cc:3037] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.362201 10743 raft_consensus.cc:3037] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.363164 10463 raft_consensus.cc:2454] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 1.
I20260924 10:45:00.363412 10743 raft_consensus.cc:2454] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 1.
I20260924 10:45:00.363458 10535 leader_election.cc:302] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.363646 10825 raft_consensus.cc:2781] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:00.363867 10825 raft_consensus.cc:686] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 1 LEADER]: Becoming Leader. State: Replica: b79e1706683247f89155284b0431393c, State: Running, Role: LEADER
I20260924 10:45:00.364187 10825 consensus_queue.cc:237] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } }
I20260924 10:45:00.367205 10306 catalog_manager.cc:5495] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c reported cstate change: term changed from 0 to 1, leader changed from <none> to b79e1706683247f89155284b0431393c (127.2.182.2). New cstate: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } health_report { overall_health: HEALTHY } } }
I20260924 10:45:00.382424 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.379376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.2.182.2:34165)
0924 10:45:00.379492 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.379495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.379507 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:45:00.379605 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.379607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.379655 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.379739 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.379935 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.380397 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.381258 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.381470 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.381557 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.381756 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.382132 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.382213 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.382267 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.382310 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:45:00.387467 10743 raft_consensus.cc:1261] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Refusing update from remote peer b79e1706683247f89155284b0431393c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.387485 10463 raft_consensus.cc:1261] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Refusing update from remote peer b79e1706683247f89155284b0431393c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.388072 10827 consensus_queue.cc:1035] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.388226 10825 consensus_queue.cc:1035] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [LEADER]: Connected to new peer: Peer: permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.395321 10830 mvcc.cc:205] Tried to move back new op lower bound from 7332850484781166592 to 7332850484694052864. Current Snapshot: MvccSnapshot[applied={T|T < 7332850484781166592}]
I20260924 10:45:00.403173 10306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.403337 10306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.405750 10306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Sending DeleteTablet for 3 replicas of tablet eebb66e6ab354adbb1d9550cf239a355
I20260924 10:45:00.406461 10443 tablet_service.cc:1514] Processing DeleteTablet for tablet eebb66e6ab354adbb1d9550cf239a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:46590
I20260924 10:45:00.406545 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet eebb66e6ab354adbb1d9550cf239a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:56664
I20260924 10:45:00.406605 10723 tablet_service.cc:1514] Processing DeleteTablet for tablet eebb66e6ab354adbb1d9550cf239a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260924 10:45:00.407056 10840 tablet_replica.cc:330] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: stopping tablet replica
I20260924 10:45:00.407354 10840 raft_consensus.cc:2229] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.407606 10841 tablet_replica.cc:330] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: stopping tablet replica
I20260924 10:45:00.407660 10306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:45:00.407902 10841 raft_consensus.cc:2229] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 10:45:00.408053 10306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:00.408195 10841 raft_consensus.cc:2258] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.408627 10840 raft_consensus.cc:2258] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.409150 10842 tablet_replica.cc:330] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: stopping tablet replica
I20260924 10:45:00.409435 10842 raft_consensus.cc:2229] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:45:00.409746 10841 ts_tablet_manager.cc:1905] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.409829 10842 raft_consensus.cc:2258] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.411315 10842 ts_tablet_manager.cc:1905] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.412520 10841 ts_tablet_manager.cc:1918] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.412662 10841 log.cc:1196] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/eebb66e6ab354adbb1d9550cf239a355
I20260924 10:45:00.413071 10841 ts_tablet_manager.cc:1939] T eebb66e6ab354adbb1d9550cf239a355 P 548d1c5a59964182be8470aa5f8a11d8: Deleting consensus metadata
I20260924 10:45:00.413456 10443 tablet_service.cc:1467] Processing CreateTablet for tablet 798662ff39654a78a68faf91f49f19a7 (DEFAULT_TABLE table=test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.413697 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798662ff39654a78a68faf91f49f19a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.413944 10842 ts_tablet_manager.cc:1918] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.414022 10842 log.cc:1196] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/eebb66e6ab354adbb1d9550cf239a355
I20260924 10:45:00.414288 10842 ts_tablet_manager.cc:1939] T eebb66e6ab354adbb1d9550cf239a355 P b79e1706683247f89155284b0431393c: Deleting consensus metadata
I20260924 10:45:00.415572 10291 catalog_manager.cc:4841] TS b79e1706683247f89155284b0431393c (127.2.182.2:34165): tablet eebb66e6ab354adbb1d9550cf239a355 (table test-table [id=88df88f84336494083cf5c4f297dfc35]) successfully deleted
I20260924 10:45:00.415927 10818 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Bootstrap starting.
I20260924 10:45:00.416134 10583 tablet_service.cc:1467] Processing CreateTablet for tablet 798662ff39654a78a68faf91f49f19a7 (DEFAULT_TABLE table=test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.416352 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798662ff39654a78a68faf91f49f19a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.416983 10818 tablet_bootstrap.cc:654] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.418378 10819 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Bootstrap starting.
I20260924 10:45:00.418496 10290 catalog_manager.cc:4841] TS 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763): tablet eebb66e6ab354adbb1d9550cf239a355 (table test-table [id=88df88f84336494083cf5c4f297dfc35]) successfully deleted
I20260924 10:45:00.418844 10723 tablet_service.cc:1467] Processing CreateTablet for tablet 798662ff39654a78a68faf91f49f19a7 (DEFAULT_TABLE table=test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.419091 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798662ff39654a78a68faf91f49f19a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.419234 10819 tablet_bootstrap.cc:654] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.419834 10840 ts_tablet_manager.cc:1905] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.420425 10819 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: No bootstrap required, opened a new log
I20260924 10:45:00.420506 10819 ts_tablet_manager.cc:1397] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:00.420960 10819 raft_consensus.cc:348] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.421063 10819 raft_consensus.cc:374] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.421097 10819 raft_consensus.cc:729] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e1706683247f89155284b0431393c, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.421209 10819 consensus_queue.cc:260] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.421471 10819 ts_tablet_manager.cc:1428] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:45:00.422089 10817 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Bootstrap starting.
I20260924 10:45:00.422269 10840 ts_tablet_manager.cc:1918] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.422350 10840 log.cc:1196] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/eebb66e6ab354adbb1d9550cf239a355
I20260924 10:45:00.422644 10840 ts_tablet_manager.cc:1939] T eebb66e6ab354adbb1d9550cf239a355 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting consensus metadata
I20260924 10:45:00.422994 10817 tablet_bootstrap.cc:654] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.423938 10818 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: No bootstrap required, opened a new log
I20260924 10:45:00.424012 10818 ts_tablet_manager.cc:1397] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260924 10:45:00.424600 10818 raft_consensus.cc:348] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.424678 10818 raft_consensus.cc:374] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.424700 10818 raft_consensus.cc:729] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff9f7a9567b49ffaae34148f21fcaf7, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.424787 10818 consensus_queue.cc:260] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.425042 10818 ts_tablet_manager.cc:1428] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:45:00.425316 10291 catalog_manager.cc:4841] TS 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613): tablet eebb66e6ab354adbb1d9550cf239a355 (table test-table [id=88df88f84336494083cf5c4f297dfc35]) successfully deleted
I20260924 10:45:00.426080 10817 tablet_bootstrap.cc:492] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: No bootstrap required, opened a new log
I20260924 10:45:00.426170 10817 ts_tablet_manager.cc:1397] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:45:00.426635 10817 raft_consensus.cc:348] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.426759 10817 raft_consensus.cc:374] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.426800 10817 raft_consensus.cc:729] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.426919 10817 consensus_queue.cc:260] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.427166 10817 ts_tablet_manager.cc:1428] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 10:45:00.458691 10650 tablet.cc:2369] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:45:00.459059 10823 raft_consensus.cc:482] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:00.459180 10823 raft_consensus.cc:504] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.460355 10823 leader_election.cc:288] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.464975 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.461087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:50003 (local address 127.2.182.2:34165)
0924 10:45:00.461195 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.461199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.461342 (+ 143us) server_negotiation.cc:407] Connection header received
0924 10:45:00.461491 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.461494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.461546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.461628 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.461961 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.462633 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.464014 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.464412 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.464533 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.464596 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.464774 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.464840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:45:00.464987 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.460866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34165 (local address 127.2.182.3:50003)
0924 10:45:00.461248 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.461265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.461378 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.461637 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.461640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.461655 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.461825 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.461832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.462789 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.462792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.463897 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.463904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.464543 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.464547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.464639 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.464661 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.464756 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.464827 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":96,"threads_started":1}
I20260924 10:45:00.465143 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.460713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:48607 (local address 127.2.182.1:40613)
0924 10:45:00.460834 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.460838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.460855 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:45:00.460927 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.460930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.460975 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.461063 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:00.461428 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.462265 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.463844 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.464450 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.464560 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.464618 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.464950 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.465025 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:45:00.465466 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.460616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40613 (local address 127.2.182.3:48607)
0924 10:45:00.460726 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.460743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.460829 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.461178 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.461182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.461200 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:00.461309 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.461315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.462386 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.462390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.463711 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.463721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.464676 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.464679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.464783 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.464804 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.464871 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.465361 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260924 10:45:00.465699 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798662ff39654a78a68faf91f49f19a7" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c" is_pre_election: true
I20260924 10:45:00.465945 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798662ff39654a78a68faf91f49f19a7" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" is_pre_election: true
I20260924 10:45:00.465978 10590 raft_consensus.cc:2454] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.466078 10463 raft_consensus.cc:2454] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.466361 10675 leader_election.cc:302] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548d1c5a59964182be8470aa5f8a11d8, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.466605 10823 raft_consensus.cc:2781] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:00.466672 10823 raft_consensus.cc:482] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:00.466714 10823 raft_consensus.cc:3037] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.467945 10823 raft_consensus.cc:504] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.468276 10823 leader_election.cc:288] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.468750 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798662ff39654a78a68faf91f49f19a7" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
I20260924 10:45:00.468791 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798662ff39654a78a68faf91f49f19a7" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c"
I20260924 10:45:00.468870 10463 raft_consensus.cc:3037] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.468924 10590 raft_consensus.cc:3037] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.469756 10463 raft_consensus.cc:2454] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.470036 10675 leader_election.cc:302] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, 548d1c5a59964182be8470aa5f8a11d8; no voters:
I20260924 10:45:00.470065 10590 raft_consensus.cc:2454] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.470237 10823 raft_consensus.cc:2781] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:00.470472 10823 raft_consensus.cc:686] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Becoming Leader. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Running, Role: LEADER
I20260924 10:45:00.470703 10823 consensus_queue.cc:237] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.472935 10305 catalog_manager.cc:5495] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "548d1c5a59964182be8470aa5f8a11d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } health_report { overall_health: HEALTHY } } }
W20260924 10:45:00.485707 10510 tablet.cc:2369] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246699, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.62@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:45:00.523439 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.514644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46211 (local address 127.0.0.1:39678)
0924 10:45:00.514928 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.514945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.515096 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.515365 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.515370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.515772 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:00.515867 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.515873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.516967 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.516970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.517539 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.517547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.518111 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.518114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.518207 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.518769 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:00.518780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:00.520110 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:00.521967 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.521975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.521979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.522184 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.522483 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:00.522485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:00.522486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:00.522556 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:00.522983 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:00.522988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:00.523112 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.523198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.523301 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":19,"thread_start_us":94,"threads_started":1}
I20260924 10:45:00.523602 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.514722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.2.182.62:46211)
0924 10:45:00.514875 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.514880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.515070 (+ 190us) server_negotiation.cc:407] Connection header received
0924 10:45:00.515171 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.515174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.515223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.515344 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.516025 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.516818 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.517668 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.518001 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.518113 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.520258 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:00.520262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:00.520265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:00.520289 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:00.521854 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.522293 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.522296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.522298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.522388 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.522645 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.522649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.522650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.522876 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:00.522975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.523289 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.523408 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":64}
I20260924 10:45:00.527736 10305 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:39678:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:45:00.528437 10305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:00.533212 10443 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb13e4491240c898daa851a18dcbab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2b0a9683dea46a9b975cc2cf103f375]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:45:00.533304 10583 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb13e4491240c898daa851a18dcbab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2b0a9683dea46a9b975cc2cf103f375]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:45:00.533625 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb13e4491240c898daa851a18dcbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.533671 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb13e4491240c898daa851a18dcbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.535897 10818 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: Bootstrap starting.
I20260924 10:45:00.536474 10819 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: Bootstrap starting.
I20260924 10:45:00.536796 10818 tablet_bootstrap.cc:654] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.537534 10819 tablet_bootstrap.cc:654] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.537463 10723 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb13e4491240c898daa851a18dcbab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2b0a9683dea46a9b975cc2cf103f375]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:45:00.537895 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb13e4491240c898daa851a18dcbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.538177 10818 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: No bootstrap required, opened a new log
I20260924 10:45:00.538276 10818 ts_tablet_manager.cc:1397] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:45:00.538743 10818 raft_consensus.cc:348] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.538857 10818 raft_consensus.cc:374] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.538905 10818 raft_consensus.cc:729] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff9f7a9567b49ffaae34148f21fcaf7, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.539031 10818 consensus_queue.cc:260] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.539317 10818 ts_tablet_manager.cc:1428] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:45:00.539289 10824 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.539422 10824 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.540867 10817 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: Bootstrap starting.
I20260924 10:45:00.541893 10817 tablet_bootstrap.cc:654] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.542210 10819 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: No bootstrap required, opened a new log
I20260924 10:45:00.542284 10819 ts_tablet_manager.cc:1397] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260924 10:45:00.542652 10819 raft_consensus.cc:348] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.542733 10819 raft_consensus.cc:374] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.542783 10819 raft_consensus.cc:729] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e1706683247f89155284b0431393c, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.542927 10819 consensus_queue.cc:260] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.543181 10817 tablet_bootstrap.cc:492] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: No bootstrap required, opened a new log
I20260924 10:45:00.543254 10817 ts_tablet_manager.cc:1397] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:45:00.543219 10836 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.543363 10836 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.543229 10819 ts_tablet_manager.cc:1428] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:45:00.543704 10817 raft_consensus.cc:348] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.543816 10817 raft_consensus.cc:374] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.543849 10817 raft_consensus.cc:729] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.543957 10817 consensus_queue.cc:260] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.544226 10823 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.544348 10823 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.544934 10817 ts_tablet_manager.cc:1428] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:45:00.559262 10836 raft_consensus.cc:482] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:00.559430 10836 raft_consensus.cc:504] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.559963 10836 leader_election.cc:288] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763)
I20260924 10:45:00.560243 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb13e4491240c898daa851a18dcbab" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" is_pre_election: true
I20260924 10:45:00.560382 10463 raft_consensus.cc:2454] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 0.
I20260924 10:45:00.560426 10743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb13e4491240c898daa851a18dcbab" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548d1c5a59964182be8470aa5f8a11d8" is_pre_election: true
I20260924 10:45:00.560580 10743 raft_consensus.cc:2454] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 0.
I20260924 10:45:00.560657 10535 leader_election.cc:302] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.560820 10836 raft_consensus.cc:2781] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:00.560870 10836 raft_consensus.cc:482] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:00.560931 10836 raft_consensus.cc:3037] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.562095 10836 raft_consensus.cc:504] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.562402 10836 leader_election.cc:288] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 election: Requested vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763)
I20260924 10:45:00.562747 10743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb13e4491240c898daa851a18dcbab" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548d1c5a59964182be8470aa5f8a11d8"
I20260924 10:45:00.562788 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb13e4491240c898daa851a18dcbab" candidate_uuid: "b79e1706683247f89155284b0431393c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
I20260924 10:45:00.562886 10463 raft_consensus.cc:3037] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.562868 10743 raft_consensus.cc:3037] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.563844 10743 raft_consensus.cc:2454] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 1.
I20260924 10:45:00.563920 10463 raft_consensus.cc:2454] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79e1706683247f89155284b0431393c in term 1.
I20260924 10:45:00.564218 10534 leader_election.cc:302] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548d1c5a59964182be8470aa5f8a11d8, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.564422 10836 raft_consensus.cc:2781] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:00.564512 10836 raft_consensus.cc:686] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [term 1 LEADER]: Becoming Leader. State: Replica: b79e1706683247f89155284b0431393c, State: Running, Role: LEADER
I20260924 10:45:00.564638 10836 consensus_queue.cc:237] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.565383 10825 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: TxnStatusTablet state changed. Reason: New leader b79e1706683247f89155284b0431393c. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.565481 10825 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:45:00.565749 10859 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:45:00.565918 10859 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:45:00.566350 10306 catalog_manager.cc:5495] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c reported cstate change: term changed from 0 to 1, leader changed from <none> to b79e1706683247f89155284b0431393c (127.2.182.2). New cstate: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } health_report { overall_health: UNKNOWN } } }
I20260924 10:45:00.582265 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.578928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34165 (local address 127.0.0.1:56680)
0924 10:45:00.579188 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.579200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.579329 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.579615 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.579617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.579625 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:45:00.579726 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.579731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.580434 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.580437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.581104 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.581114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.581547 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.581550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.581637 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.581678 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.581973 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.581989 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.582068 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.582150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:45:00.582265 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.579028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.2.182.2:34165)
0924 10:45:00.579124 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.579127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.579263 (+ 136us) server_negotiation.cc:407] Connection header received
0924 10:45:00.579448 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.579453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.579510 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.579593 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.579827 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.580309 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.581216 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.581438 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.581544 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.581810 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.581885 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.581966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.582090 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.582137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 10:45:00.584834 10743 raft_consensus.cc:1261] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Refusing update from remote peer b79e1706683247f89155284b0431393c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.584842 10463 raft_consensus.cc:1261] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Refusing update from remote peer b79e1706683247f89155284b0431393c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.585232 10836 consensus_queue.cc:1035] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.585367 10825 consensus_queue.cc:1035] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c [LEADER]: Connected to new peer: Peer: permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.586613 10823 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: TxnStatusTablet state changed. Reason: New leader b79e1706683247f89155284b0431393c. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.586719 10823 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.586719 10824 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: TxnStatusTablet state changed. Reason: New leader b79e1706683247f89155284b0431393c. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.586817 10824 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.588639 10836 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.588739 10836 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:45:00.589799 10824 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.589902 10824 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 2ff9f7a9567b49ffaae34148f21fcaf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.592341 10827 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.592437 10827 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P b79e1706683247f89155284b0431393c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:45:00.593015 10832 mvcc.cc:205] Tried to move back new op lower bound from 7332850485591724032 to 7332850485514391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332850485591724032}]
I20260924 10:45:00.593196 10823 tablet_replica.cc:439] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b79e1706683247f89155284b0431393c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } } }
I20260924 10:45:00.593300 10823 tablet_replica.cc:442] T 6fbb13e4491240c898daa851a18dcbab P 548d1c5a59964182be8470aa5f8a11d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:45:00.597254 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.592973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.2.182.3:34763)
0924 10:45:00.593067 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.593071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.593085 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:45:00.593574 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.593577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.593630 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.593726 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.594005 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.594724 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.595831 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.596061 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.596139 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.596419 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.596821 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.596972 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.597056 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.597118 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:45:00.607380 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.603663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34165 (local address 127.2.182.3:56133)
0924 10:45:00.603871 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.603886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.603959 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.604173 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.604176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.604184 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:45:00.604345 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.604351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.605168 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.605172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.606168 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.606179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.606753 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.606756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.606831 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.606872 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.607174 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.607178 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.607226 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.607283 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 10:45:00.607421 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.603759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56133 (local address 127.2.182.2:34165)
0924 10:45:00.603910 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.603914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.603926 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:45:00.604039 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.604041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.604080 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.604143 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.604469 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.605008 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.606308 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.606645 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.606742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.607012 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.607090 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.607161 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.607242 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.607295 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:45:00.613737 10463 raft_consensus.cc:1261] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.613978 10590 raft_consensus.cc:1261] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.614141 10823 consensus_queue.cc:1035] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.614446 10847 consensus_queue.cc:1035] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.618458 10831 mvcc.cc:205] Tried to move back new op lower bound from 7332850485709656064 to 7332850485130326016. Current Snapshot: MvccSnapshot[applied={T|T < 7332850485709656064}]
I20260924 10:45:00.634881 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.631468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:44113 (local address 127.2.182.3:34763)
0924 10:45:00.631558 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.631562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.631575 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:45:00.631703 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.631706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.631754 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.631819 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.632135 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.632882 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.633797 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.634071 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.634173 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.634490 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.634561 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.634619 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.634711 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.634751 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:45:00.635129 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.631155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:34763 (local address 127.2.182.2:44113)
0924 10:45:00.631475 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.631489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:00.631576 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:00.631920 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:00.631923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:00.631931 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:45:00.632027 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.632032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.632998 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.633004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.633675 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:00.633684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.634264 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.634267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:00.634333 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.634369 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.634632 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.634639 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:45:00.634698 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:45:00.634772 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260924 10:45:00.656884 10305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.657061 10305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.659224 10305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Sending DeleteTablet for 3 replicas of tablet 798662ff39654a78a68faf91f49f19a7
I20260924 10:45:00.659937 10723 tablet_service.cc:1514] Processing DeleteTablet for tablet 798662ff39654a78a68faf91f49f19a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260924 10:45:00.659925 10443 tablet_service.cc:1514] Processing DeleteTablet for tablet 798662ff39654a78a68faf91f49f19a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:46590
I20260924 10:45:00.660102 10841 tablet_replica.cc:330] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: stopping tablet replica
I20260924 10:45:00.660115 10840 tablet_replica.cc:330] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: stopping tablet replica
I20260924 10:45:00.660209 10841 raft_consensus.cc:2229] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:45:00.660231 10840 raft_consensus.cc:2229] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.660329 10840 raft_consensus.cc:2258] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.660490 10841 raft_consensus.cc:2258] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.661134 10840 ts_tablet_manager.cc:1905] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.661136 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet 798662ff39654a78a68faf91f49f19a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:56664
I20260924 10:45:00.661293 10842 tablet_replica.cc:330] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: stopping tablet replica
I20260924 10:45:00.661418 10842 raft_consensus.cc:2229] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.661540 10842 raft_consensus.cc:2258] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.661975 10841 ts_tablet_manager.cc:1905] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.663883 10840 ts_tablet_manager.cc:1918] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:45:00.663947 10840 log.cc:1196] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/798662ff39654a78a68faf91f49f19a7
I20260924 10:45:00.664191 10840 ts_tablet_manager.cc:1939] T 798662ff39654a78a68faf91f49f19a7 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting consensus metadata
I20260924 10:45:00.664604 10841 ts_tablet_manager.cc:1918] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:45:00.664664 10841 log.cc:1196] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/798662ff39654a78a68faf91f49f19a7
I20260924 10:45:00.664912 10841 ts_tablet_manager.cc:1939] T 798662ff39654a78a68faf91f49f19a7 P 548d1c5a59964182be8470aa5f8a11d8: Deleting consensus metadata
I20260924 10:45:00.664871 10291 catalog_manager.cc:4841] TS 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613): tablet 798662ff39654a78a68faf91f49f19a7 (table test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]) successfully deleted
I20260924 10:45:00.665573 10842 ts_tablet_manager.cc:1905] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.668128 10842 ts_tablet_manager.cc:1918] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:45:00.668187 10842 log.cc:1196] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/798662ff39654a78a68faf91f49f19a7
I20260924 10:45:00.668411 10842 ts_tablet_manager.cc:1939] T 798662ff39654a78a68faf91f49f19a7 P b79e1706683247f89155284b0431393c: Deleting consensus metadata
I20260924 10:45:00.669106 10291 catalog_manager.cc:4841] TS b79e1706683247f89155284b0431393c (127.2.182.2:34165): tablet 798662ff39654a78a68faf91f49f19a7 (table test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]) successfully deleted
I20260924 10:45:00.670145 10290 catalog_manager.cc:4841] TS 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763): tablet 798662ff39654a78a68faf91f49f19a7 (table test-table [id=6fb4d6b47f2649a299f56d2d1f0ba3a9]) successfully deleted
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:00.699136 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:00.691625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.2.182.62:46211)
0924 10:45:00.691755 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.691759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.692045 (+ 286us) server_negotiation.cc:407] Connection header received
0924 10:45:00.692180 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.692183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.692239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.692330 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.693228 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.693987 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.695087 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.695391 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.695499 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.699004 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39692: BlockingRecv error: recv got EOF from 127.0.0.1:39692 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM
I20260924 10:45:00.711534 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.701102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.2.182.62:46211)
0924 10:45:00.701206 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.701209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.701657 (+ 448us) server_negotiation.cc:407] Connection header received
0924 10:45:00.701766 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.701770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.701828 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.701913 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:00.702596 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.703388 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.704610 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.704980 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.705246 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.707923 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:00.707928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:00.707931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:00.707956 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:00.709730 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.710234 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.710238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.710240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.710331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:00.710631 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:00.710635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:00.710637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:00.710852 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:00.710990 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.711271 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.711376 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:45:00.714393 10305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:45:00.714972 10305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:00.719836 10443 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (DEFAULT_TABLE table=test-table [id=7a870d677d79407695e3abe317d5c36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.719957 10723 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (DEFAULT_TABLE table=test-table [id=7a870d677d79407695e3abe317d5c36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.720160 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c3fdd768a4bfdb07882e82c0a4560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.720223 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c3fdd768a4bfdb07882e82c0a4560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.722636 10818 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Bootstrap starting.
I20260924 10:45:00.723580 10818 tablet_bootstrap.cc:654] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.724191 10817 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Bootstrap starting.
I20260924 10:45:00.725095 10817 tablet_bootstrap.cc:654] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.725440 10583 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (DEFAULT_TABLE table=test-table [id=7a870d677d79407695e3abe317d5c36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.725733 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c3fdd768a4bfdb07882e82c0a4560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.725813 10818 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: No bootstrap required, opened a new log
I20260924 10:45:00.725889 10818 ts_tablet_manager.cc:1397] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 10:45:00.726300 10818 raft_consensus.cc:348] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.726411 10818 raft_consensus.cc:374] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.726439 10818 raft_consensus.cc:729] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff9f7a9567b49ffaae34148f21fcaf7, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.726473 10817 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: No bootstrap required, opened a new log
I20260924 10:45:00.726541 10817 ts_tablet_manager.cc:1397] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:00.726552 10818 consensus_queue.cc:260] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.726815 10818 ts_tablet_manager.cc:1428] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:45:00.726962 10817 raft_consensus.cc:348] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.727077 10817 raft_consensus.cc:374] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.727109 10817 raft_consensus.cc:729] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.727231 10817 consensus_queue.cc:260] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.727481 10817 ts_tablet_manager.cc:1428] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:45:00.728571 10819 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Bootstrap starting.
I20260924 10:45:00.729347 10819 tablet_bootstrap.cc:654] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.730397 10819 tablet_bootstrap.cc:492] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: No bootstrap required, opened a new log
I20260924 10:45:00.730490 10819 ts_tablet_manager.cc:1397] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:00.730943 10819 raft_consensus.cc:348] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.731055 10819 raft_consensus.cc:374] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.731083 10819 raft_consensus.cc:729] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e1706683247f89155284b0431393c, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.731215 10819 consensus_queue.cc:260] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.731451 10819 ts_tablet_manager.cc:1428] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:45:00.761149 10847 raft_consensus.cc:482] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:00.761281 10847 raft_consensus.cc:504] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.761722 10847 leader_election.cc:288] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.762135 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c3fdd768a4bfdb07882e82c0a4560" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" is_pre_election: true
I20260924 10:45:00.762142 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c3fdd768a4bfdb07882e82c0a4560" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c" is_pre_election: true
I20260924 10:45:00.762306 10463 raft_consensus.cc:2454] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.762310 10590 raft_consensus.cc:2454] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.762639 10675 leader_election.cc:302] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, 548d1c5a59964182be8470aa5f8a11d8; no voters:
I20260924 10:45:00.762821 10847 raft_consensus.cc:2781] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:00.762879 10847 raft_consensus.cc:482] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:00.762926 10847 raft_consensus.cc:3037] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.763955 10847 raft_consensus.cc:504] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.764302 10847 leader_election.cc:288] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.764616 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c3fdd768a4bfdb07882e82c0a4560" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
I20260924 10:45:00.764703 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c3fdd768a4bfdb07882e82c0a4560" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c"
I20260924 10:45:00.764763 10463 raft_consensus.cc:3037] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.764834 10590 raft_consensus.cc:3037] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.765679 10463 raft_consensus.cc:2454] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.765952 10675 leader_election.cc:302] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, 548d1c5a59964182be8470aa5f8a11d8; no voters:
I20260924 10:45:00.765992 10590 raft_consensus.cc:2454] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.766147 10847 raft_consensus.cc:2781] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:00.766232 10847 raft_consensus.cc:686] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Becoming Leader. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Running, Role: LEADER
I20260924 10:45:00.766369 10847 consensus_queue.cc:237] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.768164 10306 catalog_manager.cc:5495] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "548d1c5a59964182be8470aa5f8a11d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } health_report { overall_health: HEALTHY } } }
I20260924 10:45:00.784670 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.781344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.2.182.3:34763)
0924 10:45:00.781477 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:00.781481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:00.781495 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:45:00.781581 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:00.781583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:00.781623 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:00.781676 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:00.781962 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.782432 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.783694 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:00.783889 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:00.783957 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:00.784255 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:00.784318 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:00.784366 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:00.784486 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:45:00.784528 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:45:00.786749 10463 raft_consensus.cc:1261] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.786736 10590 raft_consensus.cc:1261] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.787201 10823 consensus_queue.cc:1035] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.787367 10847 consensus_queue.cc:1035] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.794291 10304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.794427 10304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.796751 10304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Sending DeleteTablet for 3 replicas of tablet 9b5c3fdd768a4bfdb07882e82c0a4560
I20260924 10:45:00.797415 10443 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:46590
I20260924 10:45:00.797492 10723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260924 10:45:00.797601 10840 tablet_replica.cc:330] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: stopping tablet replica
I20260924 10:45:00.797636 10841 tablet_replica.cc:330] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: stopping tablet replica
I20260924 10:45:00.797726 10841 raft_consensus.cc:2229] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:45:00.797731 10840 raft_consensus.cc:2229] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.797842 10840 raft_consensus.cc:2258] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.797885 10841 raft_consensus.cc:2258] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.798096 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c3fdd768a4bfdb07882e82c0a4560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:56664
I20260924 10:45:00.798282 10842 tablet_replica.cc:330] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: stopping tablet replica
I20260924 10:45:00.798403 10842 raft_consensus.cc:2229] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.798527 10842 raft_consensus.cc:2258] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.798637 10841 ts_tablet_manager.cc:1905] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.798635 10306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:45:00.799141 10306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:00.799340 10842 ts_tablet_manager.cc:1905] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.801075 10841 ts_tablet_manager.cc:1918] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.801126 10841 log.cc:1196] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/9b5c3fdd768a4bfdb07882e82c0a4560
I20260924 10:45:00.801299 10841 ts_tablet_manager.cc:1939] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 548d1c5a59964182be8470aa5f8a11d8: Deleting consensus metadata
I20260924 10:45:00.801753 10842 ts_tablet_manager.cc:1918] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.801817 10842 log.cc:1196] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/9b5c3fdd768a4bfdb07882e82c0a4560
I20260924 10:45:00.802073 10842 ts_tablet_manager.cc:1939] T 9b5c3fdd768a4bfdb07882e82c0a4560 P b79e1706683247f89155284b0431393c: Deleting consensus metadata
I20260924 10:45:00.802713 10840 ts_tablet_manager.cc:1905] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.802983 10290 catalog_manager.cc:4841] TS 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763): tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (table test-table [id=7a870d677d79407695e3abe317d5c36d]) successfully deleted
I20260924 10:45:00.803097 10291 catalog_manager.cc:4841] TS b79e1706683247f89155284b0431393c (127.2.182.2:34165): tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (table test-table [id=7a870d677d79407695e3abe317d5c36d]) successfully deleted
I20260924 10:45:00.805172 10840 ts_tablet_manager.cc:1918] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:00.805272 10840 log.cc:1196] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/9b5c3fdd768a4bfdb07882e82c0a4560
I20260924 10:45:00.805528 10840 ts_tablet_manager.cc:1939] T 9b5c3fdd768a4bfdb07882e82c0a4560 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting consensus metadata
I20260924 10:45:00.807220 10583 tablet_service.cc:1467] Processing CreateTablet for tablet 8946816b74114070bf5abbcb839ed739 (DEFAULT_TABLE table=test-table [id=c2441099dbcf4cef972a3a7da744c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.807284 10723 tablet_service.cc:1467] Processing CreateTablet for tablet 8946816b74114070bf5abbcb839ed739 (DEFAULT_TABLE table=test-table [id=c2441099dbcf4cef972a3a7da744c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.807471 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8946816b74114070bf5abbcb839ed739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.807570 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8946816b74114070bf5abbcb839ed739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.809276 10817 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Bootstrap starting.
I20260924 10:45:00.809446 10443 tablet_service.cc:1467] Processing CreateTablet for tablet 8946816b74114070bf5abbcb839ed739 (DEFAULT_TABLE table=test-table [id=c2441099dbcf4cef972a3a7da744c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:00.809636 10819 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Bootstrap starting.
I20260924 10:45:00.809670 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8946816b74114070bf5abbcb839ed739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:00.810165 10817 tablet_bootstrap.cc:654] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.810370 10819 tablet_bootstrap.cc:654] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.810846 10291 catalog_manager.cc:4841] TS 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613): tablet 9b5c3fdd768a4bfdb07882e82c0a4560 (table test-table [id=7a870d677d79407695e3abe317d5c36d]) successfully deleted
I20260924 10:45:00.811659 10819 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: No bootstrap required, opened a new log
I20260924 10:45:00.811754 10819 ts_tablet_manager.cc:1397] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:00.811750 10818 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: Bootstrap starting.
I20260924 10:45:00.812125 10819 raft_consensus.cc:348] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.812223 10819 raft_consensus.cc:374] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.812258 10819 raft_consensus.cc:729] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e1706683247f89155284b0431393c, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.812376 10819 consensus_queue.cc:260] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.812631 10819 ts_tablet_manager.cc:1428] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:45:00.812678 10818 tablet_bootstrap.cc:654] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:00.813617 10817 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: No bootstrap required, opened a new log
I20260924 10:45:00.813715 10817 ts_tablet_manager.cc:1397] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:45:00.814144 10817 raft_consensus.cc:348] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.814251 10817 raft_consensus.cc:374] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.814285 10817 raft_consensus.cc:729] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.814401 10817 consensus_queue.cc:260] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.814651 10817 ts_tablet_manager.cc:1428] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:45:00.815295 10818 tablet_bootstrap.cc:492] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: No bootstrap required, opened a new log
I20260924 10:45:00.815390 10818 ts_tablet_manager.cc:1397] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:45:00.815871 10818 raft_consensus.cc:348] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.815977 10818 raft_consensus.cc:374] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:00.816030 10818 raft_consensus.cc:729] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff9f7a9567b49ffaae34148f21fcaf7, State: Initialized, Role: FOLLOWER
I20260924 10:45:00.816166 10818 consensus_queue.cc:260] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.816442 10818 ts_tablet_manager.cc:1428] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:45:00.817909 10867 raft_consensus.cc:482] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:00.818013 10867 raft_consensus.cc:504] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.818354 10867 leader_election.cc:288] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.818747 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8946816b74114070bf5abbcb839ed739" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" is_pre_election: true
I20260924 10:45:00.818789 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8946816b74114070bf5abbcb839ed739" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c" is_pre_election: true
I20260924 10:45:00.818876 10463 raft_consensus.cc:2454] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.818917 10590 raft_consensus.cc:2454] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 0.
I20260924 10:45:00.819185 10675 leader_election.cc:302] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548d1c5a59964182be8470aa5f8a11d8, b79e1706683247f89155284b0431393c; no voters:
I20260924 10:45:00.819380 10867 raft_consensus.cc:2781] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:00.819449 10867 raft_consensus.cc:482] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:00.819510 10867 raft_consensus.cc:3037] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.820472 10867 raft_consensus.cc:504] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.820811 10867 leader_election.cc:288] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613), b79e1706683247f89155284b0431393c (127.2.182.2:34165)
I20260924 10:45:00.821184 10463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8946816b74114070bf5abbcb839ed739" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7"
I20260924 10:45:00.821307 10463 raft_consensus.cc:3037] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.821498 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8946816b74114070bf5abbcb839ed739" candidate_uuid: "548d1c5a59964182be8470aa5f8a11d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e1706683247f89155284b0431393c"
I20260924 10:45:00.821604 10590 raft_consensus.cc:3037] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:00.822067 10463 raft_consensus.cc:2454] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.822305 10675 leader_election.cc:302] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff9f7a9567b49ffaae34148f21fcaf7, 548d1c5a59964182be8470aa5f8a11d8; no voters:
I20260924 10:45:00.822456 10867 raft_consensus.cc:2781] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:00.822563 10867 raft_consensus.cc:686] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Becoming Leader. State: Replica: 548d1c5a59964182be8470aa5f8a11d8, State: Running, Role: LEADER
I20260924 10:45:00.822688 10590 raft_consensus.cc:2454] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548d1c5a59964182be8470aa5f8a11d8 in term 1.
I20260924 10:45:00.822675 10867 consensus_queue.cc:237] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } }
I20260924 10:45:00.824074 10304 catalog_manager.cc:5495] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "548d1c5a59964182be8470aa5f8a11d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548d1c5a59964182be8470aa5f8a11d8" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34763 } health_report { overall_health: HEALTHY } } }
I20260924 10:45:00.842607 10463 raft_consensus.cc:1261] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.842768 10590 raft_consensus.cc:1261] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Refusing update from remote peer 548d1c5a59964182be8470aa5f8a11d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:00.843241 10867 consensus_queue.cc:1035] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff9f7a9567b49ffaae34148f21fcaf7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.843432 10847 consensus_queue.cc:1035] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79e1706683247f89155284b0431393c" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:00.872534 10304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.872654 10304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:00.874347 10304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277f043b424347f992f36cbc3c6cea82: Sending DeleteTablet for 3 replicas of tablet 8946816b74114070bf5abbcb839ed739
I20260924 10:45:00.874821 10723 tablet_service.cc:1514] Processing DeleteTablet for tablet 8946816b74114070bf5abbcb839ed739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260924 10:45:00.874877 10443 tablet_service.cc:1514] Processing DeleteTablet for tablet 8946816b74114070bf5abbcb839ed739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:46590
I20260924 10:45:00.875025 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet 8946816b74114070bf5abbcb839ed739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:00 UTC) from {username='oryx', principal='oryx/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:56664
I20260924 10:45:00.875063 10840 tablet_replica.cc:330] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: stopping tablet replica
I20260924 10:45:00.875072 10841 tablet_replica.cc:330] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: stopping tablet replica
I20260924 10:45:00.875167 10841 raft_consensus.cc:2229] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:45:00.875168 10840 raft_consensus.cc:2229] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.875200 10842 tablet_replica.cc:330] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: stopping tablet replica
I20260924 10:45:00.875262 10840 raft_consensus.cc:2258] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.875316 10842 raft_consensus.cc:2229] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:00.875325 10841 raft_consensus.cc:2258] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.875422 10842 raft_consensus.cc:2258] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:00.875905 10840 ts_tablet_manager.cc:1905] T 8946816b74114070bf5abbcb839ed739 P 2ff9f7a9567b49ffaae34148f21fcaf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.876005 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 10378
I20260924 10:45:00.876138 10841 ts_tablet_manager.cc:1905] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.878126 10842 ts_tablet_manager.cc:1905] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:00.878528 10841 ts_tablet_manager.cc:1918] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:45:00.878588 10841 log.cc:1196] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/8946816b74114070bf5abbcb839ed739
I20260924 10:45:00.878818 10841 ts_tablet_manager.cc:1939] T 8946816b74114070bf5abbcb839ed739 P 548d1c5a59964182be8470aa5f8a11d8: Deleting consensus metadata
I20260924 10:45:00.879642 10290 catalog_manager.cc:4841] TS 548d1c5a59964182be8470aa5f8a11d8 (127.2.182.3:34763): tablet 8946816b74114070bf5abbcb839ed739 (table test-table [id=c2441099dbcf4cef972a3a7da744c8df]) successfully deleted
I20260924 10:45:00.880154 10842 ts_tablet_manager.cc:1918] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:45:00.880245 10842 log.cc:1196] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/8946816b74114070bf5abbcb839ed739
I20260924 10:45:00.880489 10842 ts_tablet_manager.cc:1939] T 8946816b74114070bf5abbcb839ed739 P b79e1706683247f89155284b0431393c: Deleting consensus metadata
I20260924 10:45:00.881072 10291 catalog_manager.cc:4841] TS b79e1706683247f89155284b0431393c (127.2.182.2:34165): tablet 8946816b74114070bf5abbcb839ed739 (table test-table [id=c2441099dbcf4cef972a3a7da744c8df]) successfully deleted
W20260924 10:45:00.883937 10291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:40613 (error 108)
I20260924 10:45:00.884187 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 10518
I20260924 10:45:00.885653 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:00.885146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40613 (local address 127.0.0.1:46596)
0924 10:45:00.885445 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:00.885528 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:40613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":54,"threads_started":1}
W20260924 10:45:00.885756 10291 catalog_manager.cc:4584] TS 2ff9f7a9567b49ffaae34148f21fcaf7 (127.2.182.1:40613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8946816b74114070bf5abbcb839ed739: Network error: Client connection negotiation failed: client connection to 127.2.182.1:40613: connect: Connection refused (error 111)
I20260924 10:45:00.890869 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 10658
I20260924 10:45:00.897378 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 10274
2026-09-24T10:45:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2725 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10903](info): set up 2 sockets
Sep 24 10:45:00 dist-test-slave-6lwt krb5kdc[10903](info): commencing operation
krb5kdc: starting...
W20260924 10:45:01.925737 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:45:02 dist-test-slave-6lwt krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:45:02.043744 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41585
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:41585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:02.148710 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:02.148965 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:02.149011 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:02.152990 10916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:02.153047 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:02.153066 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:02.153082 10916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:02.153097 10916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:02.158093 10916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:41585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:02.158972 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:02.159890 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:02.168057 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:02.168067 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:02.168062 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:02.168097 10916 server_base.cc:1034] running on GCE node
I20260924 10:45:02.168804 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:02.169888 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:02.171078 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790246702171066 us; error 24 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T10:45:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1282 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:45:02 dist-test-slave-6lwt krb5kdc[10929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:45:02 dist-test-slave-6lwt krb5kdc[10929](info): set up 2 sockets
Sep 24 10:45:02 dist-test-slave-6lwt krb5kdc[10929](info): commencing operation
krb5kdc: starting...
W20260924 10:45:03.209859 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:45:03.327059 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32807
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:03.433991 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:03.434265 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:03.434358 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:03.438319 10942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:03.438411 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:03.438450 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:03.438509 10942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:03.438549 10942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:03.443768 10942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:03.444761 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:03.445739 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:03.453999 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:03.454032 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:03.454156 10942 server_base.cc:1034] running on GCE node
W20260924 10:45:03.454053 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:03.454756 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:03.455762 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:03.456964 10942 hybrid_clock.cc:648] HybridClock initialized: now 1790246703456945 us; error 30 us; skew 500 ppm
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:03.463591 10942 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:45:03.464419 10942 webserver.cc:460] Webserver started at http://127.2.182.62:45107/ using document root <none> and password file <none>
I20260924 10:45:03.465023 10942 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:03.465119 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:03.465351 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:03.467317 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "4b5b0a69c67749e78e5182ebebc756b0"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "537330fa647980c2a71f3357d1a9589a"
server_key_iv: "11a8e39d44de43aa119d08f8a6e0cb9e"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.467871 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "4b5b0a69c67749e78e5182ebebc756b0"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "537330fa647980c2a71f3357d1a9589a"
server_key_iv: "11a8e39d44de43aa119d08f8a6e0cb9e"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.471585 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:45:03.473903 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.475054 10942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:03.475194 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4b5b0a69c67749e78e5182ebebc756b0"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "537330fa647980c2a71f3357d1a9589a"
server_key_iv: "11a8e39d44de43aa119d08f8a6e0cb9e"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.475301 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:03.494020 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:03.496933 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:03.497149 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:03.504546 10942 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32807
I20260924 10:45:03.504558 11009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32807 every 8 connection(s)
I20260924 10:45:03.505573 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:45:03.508252 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:03.512928 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 10942
I20260924 10:45:03.513059 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:45:03.513363 2776 external_mini_cluster.cc:1428] Setting key 79591ad04e53aae88d35197dfb8372b0
I20260924 10:45:03.514101 11010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Bootstrap starting.
I20260924 10:45:03.516501 11010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:03.517433 11010 log.cc:824] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:03.519472 11010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: No bootstrap required, opened a new log
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:03.523698 11010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:03.523909 11010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:03.524039 11010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5b0a69c67749e78e5182ebebc756b0, State: Initialized, Role: FOLLOWER
I20260924 10:45:03.524559 11010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:03.524724 11010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:45:03.524802 11010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:45:03.524927 11010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:03.525995 11010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:03.526346 11010 leader_election.cc:302] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b5b0a69c67749e78e5182ebebc756b0; no voters:
I20260924 10:45:03.526688 11010 leader_election.cc:288] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:45:03.526794 11015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:03.527223 11015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 LEADER]: Becoming Leader. State: Replica: 4b5b0a69c67749e78e5182ebebc756b0, State: Running, Role: LEADER
I20260924 10:45:03.527554 11015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:03.528002 11010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:45:03.528090 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:03.514754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.2.182.62:32807)
0924 10:45:03.515167 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:03.515184 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:03.515217 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:45:03.515802 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:03.515817 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:03.516145 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:03.516408 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:03.517322 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.518508 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.519656 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.520110 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.520224 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:03.522980 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:03.522985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:03.522998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:03.523028 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:03.525041 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:03.525446 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:03.525452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:03.525457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:03.525593 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:03.525855 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:03.525861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:03.525866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:03.526373 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:03.526572 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:03.526889 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 10:45:03.527180 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260924 10:45:03.528695 11015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5b0a69c67749e78e5182ebebc756b0. Latest consensus state: current_term: 1 leader_uuid: "4b5b0a69c67749e78e5182ebebc756b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } } }
I20260924 10:45:03.528931 11015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:03.529143 11017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b5b0a69c67749e78e5182ebebc756b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } } }
I20260924 10:45:03.529234 11017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:03.530089 11024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:45:03.533154 11024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:45:03.539036 11024 catalog_manager.cc:1347] Generated new cluster ID: 803680ea947c48369ad96a0a3e05a04b
I20260924 10:45:03.539134 11024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:45:03.549328 11024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:45:03.550422 11024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:45:03.562693 11024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Generated new TSK 0
I20260924 10:45:03.563352 11024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:45:03.596439 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:03.729741 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:03.730005 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:03.730072 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:03.734081 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:03.734130 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:03.734186 11036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:45:03.739264 11036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:03.740199 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:03.741171 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:03.750763 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:03.750748 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:03.750767 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:03.750819 11036 server_base.cc:1034] running on GCE node
I20260924 10:45:03.751428 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:03.752043 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:03.753221 11036 hybrid_clock.cc:648] HybridClock initialized: now 1790246703753207 us; error 35 us; skew 500 ppm
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:03.755867 11036 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:45:03.756711 11036 webserver.cc:460] Webserver started at http://127.2.182.1:33313/ using document root <none> and password file <none>
I20260924 10:45:03.757306 11036 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:03.757400 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:03.757624 11036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:03.759573 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "74ee361bdb8c4475a9d0e86d1c4fb185"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "9c1f56a0a6905604dcbde9de35cf816b"
server_key_iv: "38431a2828fff5fd0ad800d6168aa595"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.760136 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "74ee361bdb8c4475a9d0e86d1c4fb185"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "9c1f56a0a6905604dcbde9de35cf816b"
server_key_iv: "38431a2828fff5fd0ad800d6168aa595"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.763669 11036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:45:03.765951 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.767102 11036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:03.767230 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "74ee361bdb8c4475a9d0e86d1c4fb185"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "9c1f56a0a6905604dcbde9de35cf816b"
server_key_iv: "38431a2828fff5fd0ad800d6168aa595"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.767344 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:03.780776 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:03.784984 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:03.785193 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:03.785928 11036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:03.786782 11036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:03.786828 11036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.786870 11036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:03.786892 11036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.795396 11036 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:42067
I20260924 10:45:03.795418 11164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:42067 every 8 connection(s)
I20260924 10:45:03.796272 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:45:03.803573 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11036
I20260924 10:45:03.803712 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:45:03.803988 2776 external_mini_cluster.cc:1428] Setting key b6357c8a8cba7c2ef697c3f41fe5ab41
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:03.809636 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:03.798037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:37965 (local address 127.2.182.62:32807)
0924 10:45:03.798197 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:03.798202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:03.799191 (+ 989us) server_negotiation.cc:407] Connection header received
0924 10:45:03.800085 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:03.800088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:03.800143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:03.800247 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:03.801438 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.801957 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.803098 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.803352 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.803464 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:03.806115 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:03.806119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:03.806122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:03.806152 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:03.807603 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:03.808109 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:03.808114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:03.808116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:03.808200 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:03.808538 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:03.808541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:03.808543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:03.808715 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:03.808823 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:03.809341 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 10:45:03.809459 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:45:03.810385 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:03.798411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:37965)
0924 10:45:03.799051 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:03.799091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:03.799869 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:03.800414 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:03.800422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:03.800762 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:03.801296 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:03.801309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.802098 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.802103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:03.802958 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:03.802967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:03.803500 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:03.803505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:03.803699 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:03.804376 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:03.804392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:03.805949 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:03.807731 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:03.807737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:03.807751 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:03.807997 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:03.808293 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:03.808297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:03.808299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:03.808437 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:03.808821 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:03.808827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:03.809107 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 10:45:03.809288 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:45:03.809593 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":101,"threads_started":1}
I20260924 10:45:03.811527 11165 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32807
I20260924 10:45:03.811781 11165 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:03.812363 11165 heartbeater.cc:511] Master 127.2.182.62:32807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:45:03.813908 10974 ts_manager.cc:194] Registered new tserver with Master: 74ee361bdb8c4475a9d0e86d1c4fb185 (127.2.182.1:42067)
I20260924 10:45:03.815090 10974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:37965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:45:03.829932 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:03.935866 11170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:03.936124 11170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:03.936208 11170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:03.940213 11170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:03.940267 11170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:03.940341 11170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:45:03.945443 11170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:03.946440 11170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:03.947438 11170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:45:03.957096 11170 server_base.cc:1034] running on GCE node
W20260924 10:45:03.957052 11175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:03.957052 11176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:03.957074 11178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:03.957698 11170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:03.958470 11170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:03.959653 11170 hybrid_clock.cc:648] HybridClock initialized: now 1790246703959637 us; error 38 us; skew 500 ppm
Sep 24 10:45:03 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:03.962884 11170 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:45:03.963804 11170 webserver.cc:460] Webserver started at http://127.2.182.2:35589/ using document root <none> and password file <none>
I20260924 10:45:03.964372 11170 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:03.964468 11170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:03.964718 11170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:03.966778 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "a80e8879d6c849c388921075ebff3096"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "4fa7a89342267da04b97d0dacc3fc027"
server_key_iv: "816d256fda408f32773c121d86849092"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.967319 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "a80e8879d6c849c388921075ebff3096"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "4fa7a89342267da04b97d0dacc3fc027"
server_key_iv: "816d256fda408f32773c121d86849092"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.971057 11170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:45:03.973382 11185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.974560 11170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:03.974697 11170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "a80e8879d6c849c388921075ebff3096"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "4fa7a89342267da04b97d0dacc3fc027"
server_key_iv: "816d256fda408f32773c121d86849092"
server_key_version: "encryptionkey@0"
I20260924 10:45:03.974813 11170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:03.985152 11170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:03.987867 11170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:03.988071 11170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:03.988646 11170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:03.989540 11170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:03.989593 11170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.989667 11170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:03.989710 11170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:03.998859 11170 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:33145
I20260924 10:45:03.998878 11298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:33145 every 8 connection(s)
I20260924 10:45:03.999735 11170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:45:04.005321 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11170
I20260924 10:45:04.005458 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:45:04.005748 2776 external_mini_cluster.cc:1428] Setting key 658d82b9680c578a61bdfaf0e615ea0d
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:04.013398 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.001501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:43077 (local address 127.2.182.62:32807)
0924 10:45:04.001639 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.001642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.002431 (+ 789us) server_negotiation.cc:407] Connection header received
0924 10:45:04.003682 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.003686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.003743 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.003845 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.004953 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.005545 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.006827 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.007053 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.007173 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.009682 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:04.009691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:04.009693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:04.009714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:04.011146 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:04.011774 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:04.011776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:04.011777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:04.011847 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:04.012234 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:04.012236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:04.012237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:04.012396 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:04.012470 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:04.013174 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 10:45:04.013282 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:45:04.014549 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.001810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:43077)
0924 10:45:04.002286 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.002326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.003468 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.004005 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.004012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.004347 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.004795 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.004807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.005670 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.005675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.006706 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.006715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.007210 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.007213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.007360 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.007966 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.007983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.009526 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:04.011264 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:04.011271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:04.011289 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:04.011677 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:04.011941 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:04.011947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:04.011950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:04.012134 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:04.012501 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:04.012509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:04.012875 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 10:45:04.013134 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:45:04.013500 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
I20260924 10:45:04.015717 11299 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32807
I20260924 10:45:04.015938 11299 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:45:04.016394 11299 heartbeater.cc:511] Master 127.2.182.62:32807 requested a full tablet report, sending...
I20260924 10:45:04.017445 10974 ts_manager.cc:194] Registered new tserver with Master: a80e8879d6c849c388921075ebff3096 (127.2.182.2:33145)
I20260924 10:45:04.018067 10974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:43077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:45:04.032616 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:04.140615 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:04.140867 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:04.140975 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:04.144809 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:04.144861 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:04.144932 11304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:04.150065 11304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:04.151083 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:04.152015 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:45:04.161383 11304 server_base.cc:1034] running on GCE node
W20260924 10:45:04.161286 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.161295 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.161286 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.161880 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:04.162403 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:04.163556 11304 hybrid_clock.cc:648] HybridClock initialized: now 1790246704163546 us; error 23 us; skew 500 ppm
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:04.166338 11304 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:45:04.167121 11304 webserver.cc:460] Webserver started at http://127.2.182.3:37545/ using document root <none> and password file <none>
I20260924 10:45:04.167618 11304 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:04.167672 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:04.167821 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:04.169713 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "9040a8852e1d4796aab9b28f46c71d3f"
format_stamp: "Formatted at 2026-09-24 10:45:04 on dist-test-slave-6lwt"
server_key: "fe9217ec2553631ce6b7635804f2ea32"
server_key_iv: "41b7df38e9d4d0e417651d8393465d62"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.170195 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "9040a8852e1d4796aab9b28f46c71d3f"
format_stamp: "Formatted at 2026-09-24 10:45:04 on dist-test-slave-6lwt"
server_key: "fe9217ec2553631ce6b7635804f2ea32"
server_key_iv: "41b7df38e9d4d0e417651d8393465d62"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.173801 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:45:04.175961 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.177160 11304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:04.177273 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "9040a8852e1d4796aab9b28f46c71d3f"
format_stamp: "Formatted at 2026-09-24 10:45:04 on dist-test-slave-6lwt"
server_key: "fe9217ec2553631ce6b7635804f2ea32"
server_key_iv: "41b7df38e9d4d0e417651d8393465d62"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.177389 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:04.189411 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:04.192385 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:04.192521 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:04.193022 11304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:04.193724 11304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:04.193769 11304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.193804 11304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:04.193820 11304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.202425 11304 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:42111
I20260924 10:45:04.202447 11432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:42111 every 8 connection(s)
I20260924 10:45:04.203292 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:45:04.208276 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11304
I20260924 10:45:04.208421 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:45:04.208683 2776 external_mini_cluster.cc:1428] Setting key d4b83dc60f794936cc9d49722ed8c018
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:04.216300 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.204877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:55359 (local address 127.2.182.62:32807)
0924 10:45:04.205020 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.205024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.205790 (+ 766us) server_negotiation.cc:407] Connection header received
0924 10:45:04.207016 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.207019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.207077 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.207180 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.208342 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.209077 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.210608 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.210861 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.210939 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.213182 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:04.213189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:04.213196 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:04.213228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:04.214686 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:04.215116 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:04.215118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:04.215119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:04.215192 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:04.215469 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:04.215471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:04.215472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:04.215626 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:04.215699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:04.216102 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 10:45:04.216191 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:45:04.217067 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.205246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:55359)
0924 10:45:04.205654 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.205693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.206806 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.207344 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.207351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.207671 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.208174 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.208195 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.209204 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.209208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.210473 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.210484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.210991 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.210994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.211123 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.211567 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.211582 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.213001 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:04.214781 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:04.214785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:04.214795 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:04.215033 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:04.215270 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:04.215272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:04.215274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:04.215391 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:04.215708 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:04.215712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:04.215903 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 10:45:04.216053 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:45:04.216264 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260924 10:45:04.218104 11433 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32807
I20260924 10:45:04.218315 11433 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:04.218740 11433 heartbeater.cc:511] Master 127.2.182.62:32807 requested a full tablet report, sending...
I20260924 10:45:04.219681 10974 ts_manager.cc:194] Registered new tserver with Master: 9040a8852e1d4796aab9b28f46c71d3f (127.2.182.3:42111)
I20260924 10:45:04.220181 10974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:55359
I20260924 10:45:04.222360 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
Entry for principal oryx/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab.
WARNING: no policy specified for oryx/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
Entry for principal oryx/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab.
WARNING: no policy specified for oryx/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
Entry for principal oryx/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab.
I20260924 10:45:04.289330 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11036
I20260924 10:45:04.294839 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11170
I20260924 10:45:04.300237 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11304
I20260924 10:45:04.305158 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 10942
I20260924 10:45:04.310925 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32807
--webserver_interface=127.2.182.62
--webserver_port=45107
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:04.415831 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:04.416065 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:04.416143 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:04.420038 11442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:04.420089 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:04.420109 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:04.420167 11442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:04.420194 11442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:04.425199 11442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:04.426093 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:04.427011 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:04.434810 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.434857 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.434834 11447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.435078 11442 server_base.cc:1034] running on GCE node
I20260924 10:45:04.435755 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:04.436758 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:04.437944 11442 hybrid_clock.cc:648] HybridClock initialized: now 1790246704437913 us; error 43 us; skew 500 ppm
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:04.440464 11442 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:45:04.441372 11442 webserver.cc:460] Webserver started at http://127.2.182.62:45107/ using document root <none> and password file <none>
I20260924 10:45:04.441902 11442 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:04.442024 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:04.445621 11442 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:04.447796 11457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.448971 11442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:04.449115 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4b5b0a69c67749e78e5182ebebc756b0"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "537330fa647980c2a71f3357d1a9589a"
server_key_iv: "11a8e39d44de43aa119d08f8a6e0cb9e"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.449632 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:04.462553 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:04.465337 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:04.465546 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:04.472673 11442 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32807
I20260924 10:45:04.472699 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32807 every 8 connection(s)
I20260924 10:45:04.473696 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:45:04.476524 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11442
I20260924 10:45:04.477043 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:42067
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:32807
--webserver_port=33313
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:45:04.478164 11510 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:45:04.479822 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Bootstrap starting.
I20260924 10:45:04.495744 11510 log.cc:824] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:04.501868 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:45:04.502269 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Bootstrap complete.
I20260924 10:45:04.506361 11510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:04.506752 11510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5b0a69c67749e78e5182ebebc756b0, State: Initialized, Role: FOLLOWER
I20260924 10:45:04.507265 11510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:04.507377 11510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:45:04.507460 11510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:45:04.507576 11510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:45:04.509469 11510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:04.509784 11510 leader_election.cc:302] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b5b0a69c67749e78e5182ebebc756b0; no voters:
I20260924 10:45:04.510077 11510 leader_election.cc:288] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:45:04.510196 11514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:45:04.510426 11514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [term 2 LEADER]: Becoming Leader. State: Replica: 4b5b0a69c67749e78e5182ebebc756b0, State: Running, Role: LEADER
I20260924 10:45:04.510761 11514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } }
I20260924 10:45:04.511270 11510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:45:04.512032 11515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b5b0a69c67749e78e5182ebebc756b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } } }
I20260924 10:45:04.512151 11515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:04.512804 11516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5b0a69c67749e78e5182ebebc756b0. Latest consensus state: current_term: 2 leader_uuid: "4b5b0a69c67749e78e5182ebebc756b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5b0a69c67749e78e5182ebebc756b0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32807 } } }
I20260924 10:45:04.512917 11516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:04.513279 11523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:45:04.515836 11523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:45:04.516361 11523 catalog_manager.cc:1259] Loaded cluster ID: 803680ea947c48369ad96a0a3e05a04b
I20260924 10:45:04.516436 11523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:45:04.518071 11523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:45:04.518627 11523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b5b0a69c67749e78e5182ebebc756b0: Loaded TSK: 0
I20260924 10:45:04.519481 11523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:45:04.596040 11512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:04.596295 11512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:04.596369 11512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:04.600245 11512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:04.600332 11512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:04.600422 11512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:45:04.605630 11512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.1.keytab
--principal=oryx/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:42067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=33313
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:04.606583 11512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:04.607534 11512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:04.617588 11538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.617614 11537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.617878 11512 server_base.cc:1034] running on GCE node
W20260924 10:45:04.618088 11540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.618460 11512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:04.619181 11512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:04.620378 11512 hybrid_clock.cc:648] HybridClock initialized: now 1790246704620345 us; error 39 us; skew 500 ppm
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:04.623577 11512 init.cc:377] Logged in from keytab as oryx/127.2.182.1@KRBTEST.COM (short username oryx)
I20260924 10:45:04.624385 11512 webserver.cc:460] Webserver started at http://127.2.182.1:33313/ using document root <none> and password file <none>
I20260924 10:45:04.624959 11512 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:04.625059 11512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:04.628978 11512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:45:04.631023 11547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.632210 11512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:04.632308 11512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "74ee361bdb8c4475a9d0e86d1c4fb185"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "9c1f56a0a6905604dcbde9de35cf816b"
server_key_iv: "38431a2828fff5fd0ad800d6168aa595"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.632823 11512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:04.649612 11512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:04.652694 11512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:04.652872 11512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:04.653535 11512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:04.654251 11512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:04.654312 11512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.654373 11512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:04.654402 11512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.663632 11512 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:42067
I20260924 10:45:04.663669 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:42067 every 8 connection(s)
I20260924 10:45:04.664580 11512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:45:04.673269 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11512
I20260924 10:45:04.673853 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:33145
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:32807
--webserver_port=35589
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:04.677237 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.666733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:59411 (local address 127.2.182.62:32807)
0924 10:45:04.667204 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.667221 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.667511 (+ 290us) server_negotiation.cc:407] Connection header received
0924 10:45:04.668847 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.668857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.669159 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.669410 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.670401 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.671303 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.672330 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.672748 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.672862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.676560 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:59411: BlockingRecv error: recv got EOF from 127.2.182.1:59411 (error 108)
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":122,"threads_started":1}
I20260924 10:45:04.677309 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.666825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:59411)
0924 10:45:04.667340 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.667388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.668500 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.669491 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.669498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.669808 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.670224 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.670236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.671414 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.671417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.672227 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.672236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.672867 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.672870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.673126 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.673716 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.673740 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.676390 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":128,"threads_started":1}
W20260924 10:45:04.677898 11661 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32807 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.62:32807: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.684029 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.678333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:50387)
0924 10:45:04.678526 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.678543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.678643 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.678862 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.678867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.679039 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.679168 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.679176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.679902 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.679909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.680937 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.680951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.681363 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.681368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.681489 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.681907 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.681927 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.683867 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 10:45:04.684054 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.678392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:50387 (local address 127.2.182.62:32807)
0924 10:45:04.678504 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.678507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.678599 (+ 92us) server_negotiation.cc:407] Connection header received
0924 10:45:04.678725 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.678727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.678758 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.678818 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.679302 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.679764 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.681053 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.681252 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.681325 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.683969 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:50387: BlockingRecv error: recv got EOF from 127.2.182.1:50387 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.689179 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.684604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:55791)
0924 10:45:04.684706 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.684721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.684795 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.685066 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.685070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.685216 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.685278 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.685283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.685955 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.685960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.686465 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.686473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.686899 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.686902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.686991 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.687431 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.687441 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.689070 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 10:45:04.689278 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.684663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:55791 (local address 127.2.182.62:32807)
0924 10:45:04.684770 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.684774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.684786 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:45:04.684920 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.684923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.684967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.685052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.685386 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.685849 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.686592 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.686792 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.686897 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.689173 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:55791: BlockingRecv error: recv got EOF from 127.2.182.1:55791 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 10:45:04.689383 11661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:45:04.782294 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:04.782497 11665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:04.782532 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:04.786391 11665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:04.786438 11665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:04.786535 11665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:45:04.791601 11665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.2.keytab
--principal=oryx/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:33145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=35589
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:04.792557 11665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:04.793516 11665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:04.802158 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.802167 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.802263 11673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.802363 11665 server_base.cc:1034] running on GCE node
I20260924 10:45:04.802860 11665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:04.803450 11665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:04.804603 11665 hybrid_clock.cc:648] HybridClock initialized: now 1790246704804592 us; error 22 us; skew 500 ppm
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:04.807235 11665 init.cc:377] Logged in from keytab as oryx/127.2.182.2@KRBTEST.COM (short username oryx)
I20260924 10:45:04.808013 11665 webserver.cc:460] Webserver started at http://127.2.182.2:35589/ using document root <none> and password file <none>
I20260924 10:45:04.808486 11665 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:04.808549 11665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:04.812043 11665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:45:04.814087 11680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.815239 11665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:04.815352 11665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "a80e8879d6c849c388921075ebff3096"
format_stamp: "Formatted at 2026-09-24 10:45:03 on dist-test-slave-6lwt"
server_key: "4fa7a89342267da04b97d0dacc3fc027"
server_key_iv: "816d256fda408f32773c121d86849092"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.815836 11665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:04.827530 11665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:04.830075 11665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:04.830250 11665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:04.830762 11665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:04.831514 11665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:04.831585 11665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.831661 11665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:04.831723 11665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.840023 11665 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:33145
I20260924 10:45:04.840067 11793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:33145 every 8 connection(s)
I20260924 10:45:04.841037 11665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.850244 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11665
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.850790 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:42111
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:32807
--webserver_port=37545
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:04.851542 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.842781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:60121 (local address 127.2.182.62:32807)
0924 10:45:04.842923 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.842927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.843769 (+ 842us) server_negotiation.cc:407] Connection header received
0924 10:45:04.845015 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.845018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.845069 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.845141 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.846179 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.846689 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.847763 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.847962 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.848041 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.851442 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:60121: BlockingRecv error: recv got EOF from 127.2.182.2:60121 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:45:04.852322 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.843214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:60121)
0924 10:45:04.843624 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.843665 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.844749 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.845267 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.845273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.845606 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.846058 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.846070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.846789 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.846791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.847662 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.847671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.848095 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.848098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.848227 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.848823 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.848839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.851335 (+ 2496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
W20260924 10:45:04.852816 11794 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32807 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.62:32807: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.858817 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.853230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:33223)
0924 10:45:04.853416 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.853429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.853497 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.853726 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.853728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.853844 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.854010 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.854018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.855052 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.855054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.855671 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.855679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.856327 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.856330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.856392 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.856882 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.856929 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.858736 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260924 10:45:04.859081 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.853346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33223 (local address 127.2.182.62:32807)
0924 10:45:04.853478 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.853483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.853498 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:45:04.853567 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.853569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.853620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.853713 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.854141 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.854940 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.855793 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.856210 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.856330 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.858913 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:33223: BlockingRecv error: recv got EOF from 127.2.182.2:33223 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:04.864267 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:04.859333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:52393)
0924 10:45:04.859429 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:04.859443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:04.859506 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:04.859763 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:04.859764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:04.859874 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:04.859932 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.859936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.860986 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.860989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.861452 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:04.861458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.862067 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.862069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:04.862119 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.862554 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:04.862564 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:04.864191 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:45:04.864426 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:04.859431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:52393 (local address 127.2.182.62:32807)
0924 10:45:04.859548 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:04.859552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:04.859567 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:45:04.859603 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:04.859609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:04.859653 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:04.859737 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:04.860039 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.860824 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.861561 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:04.861942 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:04.862075 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:04.864330 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:52393: BlockingRecv error: recv got EOF from 127.2.182.2:52393 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 10:45:04.864445 11794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:45:04.960865 11797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:04.961138 11797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:04.961180 11797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:04.966390 11797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:04.966454 11797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:04.966518 11797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:04.971871 11797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/krb5kdc/oryx_127.2.182.3.keytab
--principal=oryx/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=37545
--tserver_master_addrs=127.2.182.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:04.972841 11797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:04.973866 11797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:04.983282 11803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:04.983302 11805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.983330 11797 server_base.cc:1034] running on GCE node
W20260924 10:45:04.983282 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:04.984056 11797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:04.984606 11797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:04.985781 11797 hybrid_clock.cc:648] HybridClock initialized: now 1790246704985736 us; error 57 us; skew 500 ppm
Sep 24 10:45:04 dist-test-slave-6lwt krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246704, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:04.988343 11797 init.cc:377] Logged in from keytab as oryx/127.2.182.3@KRBTEST.COM (short username oryx)
I20260924 10:45:04.989223 11797 webserver.cc:460] Webserver started at http://127.2.182.3:37545/ using document root <none> and password file <none>
I20260924 10:45:04.989759 11797 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:04.989861 11797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:04.993516 11797 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:04.995462 11812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:04.996675 11797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:04.996801 11797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "9040a8852e1d4796aab9b28f46c71d3f"
format_stamp: "Formatted at 2026-09-24 10:45:04 on dist-test-slave-6lwt"
server_key: "fe9217ec2553631ce6b7635804f2ea32"
server_key_iv: "41b7df38e9d4d0e417651d8393465d62"
server_key_version: "encryptionkey@0"
I20260924 10:45:04.997375 11797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:05.025870 11797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:05.028558 11797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:05.028712 11797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:05.029300 11797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:05.030035 11797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:05.030081 11797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:05.030117 11797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:05.030138 11797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:05.038893 11797 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:42111
I20260924 10:45:05.038926 11925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:42111 every 8 connection(s)
I20260924 10:45:05.039793 11797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:45:05.048605 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11797
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:05.050591 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:05.041471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56991 (local address 127.2.182.62:32807)
0924 10:45:05.041611 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.041614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.042410 (+ 796us) server_negotiation.cc:407] Connection header received
0924 10:45:05.043708 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.043712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.043770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.043875 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.045067 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.045589 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.046723 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.046952 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.047071 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.050474 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:56991: BlockingRecv error: recv got EOF from 127.2.182.3:56991 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:45:05.051347 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.041801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:56991)
0924 10:45:05.042247 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:05.042288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:05.043478 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:05.044048 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:05.044055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:05.044383 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:05.044880 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.044893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.045716 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.045720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.046595 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.046604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.047109 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.047113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.047278 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.047861 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:05.047876 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:05.050375 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":102,"threads_started":1}
W20260924 10:45:05.051882 11926 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32807 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.62:32807: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:05.057935 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.052242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:40987)
0924 10:45:05.052417 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:05.052430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:05.052498 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:05.052693 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:05.052695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:05.052856 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:05.052995 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.053000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.053935 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.053940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.054718 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.054729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.055296 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.055299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.055366 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.055820 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:05.055835 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:05.057803 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 10:45:05.058010 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:05.052366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:40987 (local address 127.2.182.62:32807)
0924 10:45:05.052474 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.052478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.052491 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:45:05.052559 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.052562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.052598 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.052670 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.053109 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.053831 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.054849 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.055182 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.055297 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.057904 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:40987: BlockingRecv error: recv got EOF from 127.2.182.3:40987 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:45:05.060309 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.049624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.2.182.62:32807)
0924 10:45:05.049847 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.049851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.050850 (+ 999us) server_negotiation.cc:407] Connection header received
0924 10:45:05.050957 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.050959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.050996 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.051073 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.051938 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.052696 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.053959 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.054300 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.054405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.055888 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:05.055897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:05.055917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:05.055946 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:05.057963 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:05.058653 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:05.058660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:05.058669 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:05.058824 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:05.059124 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:05.059127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:05.059129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:05.059513 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:05.059652 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:05.059906 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:45:05.060157 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:05.063441 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.058380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:44821)
0924 10:45:05.058639 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:05.058652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:05.058734 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:05.058960 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:05.058963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:05.059115 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:05.059204 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.059210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.059978 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.059982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.060656 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.060665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.061188 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.061190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.061266 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.061636 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:05.061652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:05.063367 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260924 10:45:05.063578 11663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:05.058486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:44821 (local address 127.2.182.62:32807)
0924 10:45:05.058643 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.058647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.058693 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:45:05.058812 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.058813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.058844 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.058932 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.059311 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.059875 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.060764 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.061088 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.061191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.063467 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:44821: BlockingRecv error: recv got EOF from 127.2.182.3:44821 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 10:45:05.063637 11926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:05.695037 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.690066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:60757)
0924 10:45:05.690339 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:05.690356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:05.690490 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:05.690768 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:05.690770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:05.690935 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:05.691054 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.691060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.691745 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.691747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.692292 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.692299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.692703 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.692706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.692773 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.693225 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:05.693236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:05.694911 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
W20260924 10:45:05.695160 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:05.690124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:60757 (local address 127.2.182.62:32807)
0924 10:45:05.690371 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.690375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.690441 (+ 66us) server_negotiation.cc:407] Connection header received
0924 10:45:05.690617 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.690619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.690662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.690733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.691149 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.691660 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.692394 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.692612 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.692683 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.695034 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:60757: BlockingRecv error: recv got EOF from 127.2.182.1:60757 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:05 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:05.870000 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:05.864972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:33289)
0924 10:45:05.865232 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:05.865245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:05.865338 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:05.865569 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:05.865571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:05.865740 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:05.865831 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.865837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.866534 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.866537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.867078 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:05.867085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.867474 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.867477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:05.867536 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.868074 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:05.868085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:05.869875 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
W20260924 10:45:05.870134 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:05.865056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33289 (local address 127.2.182.62:32807)
0924 10:45:05.865241 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:05.865247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:05.865325 (+ 78us) server_negotiation.cc:407] Connection header received
0924 10:45:05.865419 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:05.865422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:05.865463 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:05.865537 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:05.865936 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.866444 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.867177 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:05.867381 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:05.867454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:05.869994 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:33289: BlockingRecv error: recv got EOF from 127.2.182.2:33289 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:06.070457 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:06.064219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:42839)
0924 10:45:06.064658 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:06.064674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:06.064825 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:06.065174 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:06.065177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:06.065433 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:06.065557 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.065564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.066614 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.066618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.067334 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.067344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.067758 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.067761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.067830 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.068350 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:06.068362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:06.070320 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":83,"threads_started":1}
W20260924 10:45:06.070623 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:06.064442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:42839 (local address 127.2.182.62:32807)
0924 10:45:06.064627 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:06.064632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:06.064771 (+ 139us) server_negotiation.cc:407] Connection header received
0924 10:45:06.064986 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:06.064992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:06.065046 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:06.065144 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:06.065699 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.066485 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.067447 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.067656 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.067734 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.070488 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:42839: BlockingRecv error: recv got EOF from 127.2.182.3:42839 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:06.701047 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:06.695843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:51673)
0924 10:45:06.696110 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:06.696126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:06.696230 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:06.696476 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:06.696478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:06.696667 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:06.696762 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.696768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.697551 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.697553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.698101 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.698107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.698508 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.698511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.698581 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.699061 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:06.699073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:06.700855 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
W20260924 10:45:06.701236 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:06.695928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:51673 (local address 127.2.182.62:32807)
0924 10:45:06.696168 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:06.696171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:06.696184 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:45:06.696319 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:06.696321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:06.696363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:06.696437 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:06.696861 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.697448 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.698198 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.698415 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.698489 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.701103 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:51673: BlockingRecv error: recv got EOF from 127.2.182.1:51673 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:06 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:06.875909 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:06.870802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:33105)
0924 10:45:06.871062 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:06.871077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:06.871186 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:06.871418 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:06.871421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:06.871616 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:06.871712 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.871717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.872419 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.872421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.873041 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:06.873050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.873489 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.873493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:06.873566 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.874081 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:06.874093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:06.875770 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260924 10:45:06.875986 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:06.870869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33105 (local address 127.2.182.62:32807)
0924 10:45:06.871024 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:06.871028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:06.871162 (+ 134us) server_negotiation.cc:407] Connection header received
0924 10:45:06.871266 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:06.871268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:06.871311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:06.871381 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:06.871809 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.872329 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.873140 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:06.873350 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:06.873429 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:06.875895 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:33105: BlockingRecv error: recv got EOF from 127.2.182.2:33105 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:07.076635 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:07.071355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:55471)
0924 10:45:07.071709 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:07.071725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:07.071840 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:07.072123 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:07.072126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:07.072373 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:07.072470 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.072476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.073245 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.073248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.073809 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.073816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.074199 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.074201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.074261 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.074718 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:07.074729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:07.076491 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
W20260924 10:45:07.076781 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:07.071525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:55471 (local address 127.2.182.62:32807)
0924 10:45:07.071652 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:07.071656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:07.071820 (+ 164us) server_negotiation.cc:407] Connection header received
0924 10:45:07.071943 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:07.071947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:07.071998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:07.072076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:07.072612 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.073149 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.073902 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.074107 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.074180 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.076628 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:55471: BlockingRecv error: recv got EOF from 127.2.182.3:55471 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:07.707692 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:07.701974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:52421)
0924 10:45:07.702267 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:07.702281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:07.702373 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:07.702616 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:07.702618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:07.702828 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:07.702923 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.702929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.703904 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.703908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.704594 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.704604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.705112 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.705118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.705200 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.705772 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:07.705784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:07.707579 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260924 10:45:07.707876 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:07.702031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:52421 (local address 127.2.182.62:32807)
0924 10:45:07.702303 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:07.702307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:07.702323 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:45:07.702461 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:07.702463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:07.702505 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:07.702583 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:07.703056 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.703771 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.704706 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.704985 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.705099 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.707725 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:52421: BlockingRecv error: recv got EOF from 127.2.182.1:52421 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:07 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:07.882303 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:07.876656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:49141)
0924 10:45:07.876975 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:07.876996 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:07.877129 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:07.877421 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:07.877424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:07.877643 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:07.877733 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.877738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.878504 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.878510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.879068 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:07.879075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.879540 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.879543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:07.879630 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.880136 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:07.880149 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:07.882107 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
W20260924 10:45:07.882331 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:07.876755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:49141 (local address 127.2.182.62:32807)
0924 10:45:07.876951 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:07.876956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:07.877066 (+ 110us) server_negotiation.cc:407] Connection header received
0924 10:45:07.877226 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:07.877231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:07.877280 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:07.877374 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:07.877861 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.878398 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.879186 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:07.879429 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:07.879538 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:07.882233 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:49141: BlockingRecv error: recv got EOF from 127.2.182.2:49141 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:08.082856 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:08.077448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:57443 (local address 127.2.182.62:32807)
0924 10:45:08.077655 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:08.077658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:08.077731 (+ 73us) server_negotiation.cc:407] Connection header received
0924 10:45:08.077836 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:08.077838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:08.077884 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:08.077950 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:08.078343 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.078860 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.079556 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.079759 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.079830 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.082766 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:57443: BlockingRecv error: recv got EOF from 127.2.182.3:57443 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 10:45:08.082859 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:08.077351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:57443)
0924 10:45:08.077649 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:08.077662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:08.077760 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:08.077986 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:08.077988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:08.078161 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:08.078254 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.078259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.078945 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.078947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.079463 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.079469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.079850 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.079853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.079912 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.080398 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:08.080409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:08.082649 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:08.719334 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:08.708598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:51361)
0924 10:45:08.709074 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:08.709091 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:08.709237 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:08.709566 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:08.709569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:08.709748 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:08.709845 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.709850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.710559 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.710561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.711112 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.711119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.711697 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.711701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.711766 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.717236 (+ 5470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:08.717251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:08.719215 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":88,"threads_started":1}
W20260924 10:45:08.719544 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:08.708638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:51361 (local address 127.2.182.62:32807)
0924 10:45:08.708975 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:08.708978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:08.709173 (+ 195us) server_negotiation.cc:407] Connection header received
0924 10:45:08.709385 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:08.709389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:08.709449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:08.709523 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:08.709940 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.710459 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.711209 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.711523 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.711678 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.719383 (+ 7705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:51361: BlockingRecv error: recv got EOF from 127.2.182.1:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":89,"threads_started":1}
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:08 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:08.888329 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:08.883259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:38149)
0924 10:45:08.883505 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:08.883528 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:08.883627 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:08.883862 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:08.883864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:08.884052 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:08.884151 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.884155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.884850 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.884852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.885423 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:08.885431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.885840 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.885843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:08.885901 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.886409 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:08.886420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:08.888214 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
W20260924 10:45:08.888464 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:08.883309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38149 (local address 127.2.182.62:32807)
0924 10:45:08.883473 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:08.883477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:08.883604 (+ 127us) server_negotiation.cc:407] Connection header received
0924 10:45:08.883709 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:08.883711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:08.883754 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:08.883821 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:08.884247 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.884761 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.885524 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:08.885743 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:08.885819 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:08.888343 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:38149: BlockingRecv error: recv got EOF from 127.2.182.2:38149 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:09.089362 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:09.083546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:49761)
0924 10:45:09.083869 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:09.083884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:09.084025 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:09.084309 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:09.084312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:09.084524 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:09.084631 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.084636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.085475 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.085480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.086048 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.086056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.086521 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.086524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.086618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.087244 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:09.087258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:09.089191 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":87,"threads_started":1}
W20260924 10:45:09.089421 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:09.083676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:49761 (local address 127.2.182.62:32807)
0924 10:45:09.083840 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:09.083845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:09.083969 (+ 124us) server_negotiation.cc:407] Connection header received
0924 10:45:09.084134 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:09.084139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:09.084195 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:09.084282 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:09.084753 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.085363 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.086171 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.086402 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.086516 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.089312 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:49761: BlockingRecv error: recv got EOF from 127.2.182.3:49761 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:09.726135 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:09.720271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:41521)
0924 10:45:09.720547 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:09.720560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:09.720647 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:09.720964 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:09.720966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:09.721146 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:09.721240 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.721245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.722018 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.722021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.722623 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.722632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.723049 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.723051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.723125 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.723736 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:09.723754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:09.725951 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
W20260924 10:45:09.726258 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:09.720347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:41521 (local address 127.2.182.62:32807)
0924 10:45:09.720603 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:09.720607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:09.720621 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:45:09.720740 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:09.720742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:09.720782 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:09.720882 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:09.721341 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.721870 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.722734 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.722947 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.723020 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.726119 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:41521: BlockingRecv error: recv got EOF from 127.2.182.1:41521 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:09 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:09.894755 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:09.889380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:55709)
0924 10:45:09.889714 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:09.889731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:09.889843 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:09.890109 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:09.890112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:09.890330 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:09.890453 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.890459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.891218 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.891224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.891826 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:09.891834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.892279 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.892282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:09.892375 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.892837 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:09.892847 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:09.894587 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
W20260924 10:45:09.894795 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:09.889564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:55709 (local address 127.2.182.62:32807)
0924 10:45:09.889716 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:09.889722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:09.889793 (+ 71us) server_negotiation.cc:407] Connection header received
0924 10:45:09.889950 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:09.889953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:09.890008 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:09.890092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:09.890570 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.891101 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.891944 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:09.892171 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:09.892276 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:09.894696 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:55709: BlockingRecv error: recv got EOF from 127.2.182.2:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:10.096032 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:10.090219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:36223)
0924 10:45:10.090538 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:10.090562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:10.090682 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:10.090957 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:10.090960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:10.091194 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:10.091278 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.091284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.092055 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.092058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.093030 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.093058 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.093514 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.093517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.093585 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.094059 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:10.094071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:10.095870 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
W20260924 10:45:10.096087 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:10.090274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:36223 (local address 127.2.182.62:32807)
0924 10:45:10.090425 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:10.090429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:10.090639 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:45:10.090789 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:10.090793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:10.090847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:10.090933 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:10.091399 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.091926 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.093156 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.093403 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.093487 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.095979 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:36223: BlockingRecv error: recv got EOF from 127.2.182.3:36223 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:10.731957 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:10.726874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:35531)
0924 10:45:10.727139 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:10.727153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:10.727242 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:10.727477 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:10.727479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:10.727658 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:10.727751 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.727757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.728578 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.728580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.729134 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.729142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.729564 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.729566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.729625 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.730114 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:10.730126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:10.731841 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20260924 10:45:10.732133 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:10.726953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:35531 (local address 127.2.182.62:32807)
0924 10:45:10.727227 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:10.727230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:10.727243 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:45:10.727325 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:10.727328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:10.727369 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:10.727442 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:10.727901 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.728470 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.729234 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.729470 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.729551 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.731970 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:35531: BlockingRecv error: recv got EOF from 127.2.182.1:35531 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:10 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:10.901074 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:10.895534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:56785)
0924 10:45:10.895872 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:10.895894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:10.896030 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:10.896293 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:10.896296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:10.896508 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:10.896615 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.896621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.897458 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.897463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.898030 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:10.898038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.898473 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.898477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:10.898569 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.899093 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:10.899105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:10.900875 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
W20260924 10:45:10.901230 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:10.895678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:56785 (local address 127.2.182.62:32807)
0924 10:45:10.895833 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:10.895837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:10.895967 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:45:10.896127 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:10.896132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:10.896184 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:10.896270 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:10.896752 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.897347 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.898146 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:10.898364 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:10.898470 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:10.901083 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:56785: BlockingRecv error: recv got EOF from 127.2.182.2:56785 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:11.102144 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:11.096781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:50401)
0924 10:45:11.097090 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:11.097106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:11.097227 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:11.097459 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:11.097461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:11.097652 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:11.097743 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.097748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.098446 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.098448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.098994 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.099000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.099389 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.099392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.099456 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.099987 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:11.100003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:11.101977 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
W20260924 10:45:11.102208 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:11.096840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:50401 (local address 127.2.182.62:32807)
0924 10:45:11.097024 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:11.097029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:11.097188 (+ 159us) server_negotiation.cc:407] Connection header received
0924 10:45:11.097309 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:11.097312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:11.097359 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:11.097423 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:11.097840 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.098360 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.099086 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.099294 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.099367 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.102102 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:50401: BlockingRecv error: recv got EOF from 127.2.182.3:50401 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:11.738510 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:11.732737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:39719)
0924 10:45:11.733075 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:11.733089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:11.733188 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:11.733399 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:11.733401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:11.733581 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:11.733675 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.733680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.734379 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.734382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.735101 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.735116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.735689 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.735693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.735782 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.736406 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:11.736418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:11.738396 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
W20260924 10:45:11.738685 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:11.732808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:39719 (local address 127.2.182.62:32807)
0924 10:45:11.733058 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:11.733061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:11.733157 (+ 96us) server_negotiation.cc:407] Connection header received
0924 10:45:11.733266 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:11.733268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:11.733311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:11.733390 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:11.733768 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.734287 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.735219 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.735556 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.735673 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.738547 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:39719: BlockingRecv error: recv got EOF from 127.2.182.1:39719 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:11 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:45:11.907454 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:11.901852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:39671 (local address 127.2.182.62:32807)
0924 10:45:11.902010 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:11.902015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:11.902266 (+ 251us) server_negotiation.cc:407] Connection header received
0924 10:45:11.902397 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:11.902400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:11.902442 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:11.902522 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:11.902949 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.903479 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.904286 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.904507 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.904622 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.907364 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:39671: BlockingRecv error: recv got EOF from 127.2.182.2:39671 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:45:11.907477 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:11.901782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:39671)
0924 10:45:11.902162 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:11.902174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:11.902301 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:11.902541 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:11.902544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:11.902750 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:11.902835 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.902841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.903595 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.903602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.904168 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:11.904176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:11.904625 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:11.904629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:11.904715 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:11.905263 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:11.905284 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:11.907259 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":83,"threads_started":1}
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:12.108381 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:12.102964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:42189)
0924 10:45:12.103322 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:12.103336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:12.103443 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:12.103676 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:12.103678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:12.103858 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:12.103951 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.103956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.104638 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.104641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.105242 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.105251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.105633 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.105635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.105692 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.106159 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:12.106170 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:12.108215 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":105,"threads_started":1}
W20260924 10:45:12.108491 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:12.103031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:42189 (local address 127.2.182.62:32807)
0924 10:45:12.103208 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:12.103212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:12.103407 (+ 195us) server_negotiation.cc:407] Connection header received
0924 10:45:12.103519 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:12.103522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:12.103569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:12.103635 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:12.104039 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.104550 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.105340 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.105546 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.105613 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.108349 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:42189: BlockingRecv error: recv got EOF from 127.2.182.3:42189 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:12.744771 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:12.739368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:49029)
0924 10:45:12.739640 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:12.739655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:12.739750 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:12.739974 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:12.739977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:12.740158 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:12.740249 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.740254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.740979 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.740983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.741529 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.741537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.742076 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.742079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.742150 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.742668 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:12.742682 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:12.744651 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
W20260924 10:45:12.744947 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:12.739514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:49029 (local address 127.2.182.62:32807)
0924 10:45:12.739752 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:12.739755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:12.739767 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:45:12.739817 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:12.739819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:12.739859 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:12.739934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:12.740343 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.740846 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.741683 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.741930 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.742019 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.744791 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:49029: BlockingRecv error: recv got EOF from 127.2.182.1:49029 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:12 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:12.913920 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:12.908286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:48127)
0924 10:45:12.908558 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:12.908571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:12.908698 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:12.909019 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:12.909022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:12.909285 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:12.909416 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.909429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.910275 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.910277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.910894 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:12.910903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.911303 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.911305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:12.911362 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.911873 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:12.911889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:12.913806 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
W20260924 10:45:12.914103 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:12.908340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48127 (local address 127.2.182.62:32807)
0924 10:45:12.908479 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:12.908483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:12.908628 (+ 145us) server_negotiation.cc:407] Connection header received
0924 10:45:12.908796 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:12.908799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:12.908875 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:12.908977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:12.909524 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.910185 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.910987 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:12.911209 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:12.911287 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:12.913938 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:48127: BlockingRecv error: recv got EOF from 127.2.182.2:48127 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:13.115268 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:13.109366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:60307)
0924 10:45:13.109717 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:13.109731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:13.109867 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:13.110150 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:13.110154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:13.110375 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:13.110465 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.110471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.111259 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.111265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.111840 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.111848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.112330 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.112334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.112437 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.113083 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:13.113096 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:13.115158 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
W20260924 10:45:13.115401 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:13.109440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:60307 (local address 127.2.182.62:32807)
0924 10:45:13.109605 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:13.109610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:13.109805 (+ 195us) server_negotiation.cc:407] Connection header received
0924 10:45:13.109980 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:13.109982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:13.110039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:13.110126 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:13.110585 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.111116 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.111956 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.112216 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.112324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.115291 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:60307: BlockingRecv error: recv got EOF from 127.2.182.3:60307 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:13.756321 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:13.745648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:42759)
0924 10:45:13.745938 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:13.745952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:13.746043 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:13.746252 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:13.746254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:13.746426 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:13.746516 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.746521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.747237 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.747239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.747797 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.747805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.748206 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.748208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.748268 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.754334 (+ 6066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:13.754347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:13.756191 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
W20260924 10:45:13.756508 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:13.745833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:42759 (local address 127.2.182.62:32807)
0924 10:45:13.746054 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:13.746057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:13.746072 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:45:13.746105 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:13.746107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:13.746146 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:13.746220 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:13.746611 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.747147 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.747892 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.748117 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.748186 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.756339 (+ 8153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:42759: BlockingRecv error: recv got EOF from 127.2.182.1:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:13 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:13.920073 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:13.914674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:60555)
0924 10:45:13.914976 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:13.914989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:13.915106 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:13.915317 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:13.915319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:13.915494 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:13.915588 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.915593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.916282 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.916285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.916817 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:13.916824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.917272 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.917276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:13.917343 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.917846 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:13.917857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:13.919959 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
W20260924 10:45:13.920267 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:13.914753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:60555 (local address 127.2.182.62:32807)
0924 10:45:13.914903 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:13.914907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:13.915055 (+ 148us) server_negotiation.cc:407] Connection header received
0924 10:45:13.915171 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:13.915173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:13.915215 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:13.915278 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:13.915691 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.916192 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.916936 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:13.917150 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:13.917228 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:13.920137 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:60555: BlockingRecv error: recv got EOF from 127.2.182.2:60555 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:14.121790 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:14.116049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:33797)
0924 10:45:14.116284 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:14.116303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:14.116419 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:14.116716 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:14.116719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:14.117013 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:14.117143 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.117151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.118048 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.118051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.118701 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.118710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.119124 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.119127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.119205 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.119769 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:14.119790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:14.121629 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
W20260924 10:45:14.121851 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:14.116090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:33797 (local address 127.2.182.62:32807)
0924 10:45:14.116210 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:14.116213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:14.116379 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:45:14.116507 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:14.116509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:14.116582 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:14.116670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:14.117267 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.117948 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.118812 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.119029 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.119109 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.121747 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:33797: BlockingRecv error: recv got EOF from 127.2.182.3:33797 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:14.762177 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:14.757125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:53679)
0924 10:45:14.757386 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:14.757400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:14.757491 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:14.757702 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:14.757704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:14.757881 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:14.757974 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.757979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.758678 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.758680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.759209 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.759215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.759613 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.759616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.759676 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.760249 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:14.760261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:14.762069 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
W20260924 10:45:14.762334 11968 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:14.757159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:53679 (local address 127.2.182.62:32807)
0924 10:45:14.757396 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:14.757399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:14.757452 (+ 53us) server_negotiation.cc:407] Connection header received
0924 10:45:14.757555 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:14.757557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:14.757598 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:14.757672 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:14.758079 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.758590 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.759308 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.759516 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.759585 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.762182 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:53679: BlockingRecv error: recv got EOF from 127.2.182.1:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:14 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:14.926366 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:14.920810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:32823)
0924 10:45:14.921099 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:14.921111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:14.921204 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:14.921429 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:14.921431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:14.921597 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:14.921690 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.921695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.922418 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.922422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.922967 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:14.922975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.923438 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.923442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:14.923536 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.924193 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:14.924212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:14.926233 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
W20260924 10:45:14.926476 11968 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:14.920869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:32823 (local address 127.2.182.62:32807)
0924 10:45:14.921028 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:14.921031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:14.921177 (+ 146us) server_negotiation.cc:407] Connection header received
0924 10:45:14.921271 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:14.921273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:14.921313 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:14.921382 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:14.921780 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.922304 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.923067 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:14.923316 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:14.923394 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:14.926350 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:32823: BlockingRecv error: recv got EOF from 127.2.182.2:32823 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:15.127864 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:15.122662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:56439)
0924 10:45:15.122952 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:15.122965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:15.123059 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:15.123309 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:15.123312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:15.123488 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:15.123585 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.123590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.124280 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.124282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.124826 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.124833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.125265 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.125268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.125327 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.125885 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:15.125897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:15.127723 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":119,"threads_started":1}
W20260924 10:45:15.128007 11968 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:15.122769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56439 (local address 127.2.182.62:32807)
0924 10:45:15.122912 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:15.122915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:15.123044 (+ 129us) server_negotiation.cc:407] Connection header received
0924 10:45:15.123155 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:15.123158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:15.123201 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:15.123269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:15.123683 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.124190 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.124954 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.125167 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.125244 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.127869 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:56439: BlockingRecv error: recv got EOF from 127.2.182.3:56439 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:15.768000 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:15.762890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:32981)
0924 10:45:15.763184 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:15.763198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:15.763286 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:15.763534 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:15.763536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:15.763718 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:15.763812 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.763818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.764530 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.764532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.765090 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.765099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.765480 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.765483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.765560 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.766044 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:15.766055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:15.767886 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260924 10:45:15.768174 11972 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:15.762977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:32981 (local address 127.2.182.62:32807)
0924 10:45:15.763248 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:15.763252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:15.763271 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:45:15.763372 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:15.763374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:15.763418 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:15.763499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:15.763909 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.764445 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.765182 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.765388 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.765461 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.768034 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:32981: BlockingRecv error: recv got EOF from 127.2.182.1:32981 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:15 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:15.932710 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:15.927165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:46507)
0924 10:45:15.927449 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:15.927462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:15.927560 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:15.927791 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:15.927793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:15.927969 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:15.928062 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.928067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.928777 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.928779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.929353 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:15.929360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.929970 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.929973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:15.930034 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.930590 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:15.930601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:15.932512 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
W20260924 10:45:15.932746 11972 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:15.927221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:46507 (local address 127.2.182.62:32807)
0924 10:45:15.927372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:15.927376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:15.927536 (+ 160us) server_negotiation.cc:407] Connection header received
0924 10:45:15.927642 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:15.927644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:15.927689 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:15.927756 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:15.928168 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.928689 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.929616 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:15.929839 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:15.929962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:15.932642 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:46507: BlockingRecv error: recv got EOF from 127.2.182.2:46507 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:16.133749 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:16.128702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:43719)
0924 10:45:16.129036 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:16.129051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:16.129194 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:16.129436 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:16.129439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:16.129618 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:16.129711 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.129717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.130379 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.130381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.130910 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.130917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.131294 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.131296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.131354 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.131794 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:16.131805 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:16.133617 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":118,"threads_started":1}
W20260924 10:45:16.133873 11972 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:16.128732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:43719 (local address 127.2.182.62:32807)
0924 10:45:16.128886 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:16.128891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:16.129160 (+ 269us) server_negotiation.cc:407] Connection header received
0924 10:45:16.129284 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:16.129286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:16.129335 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:16.129401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:16.129806 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.130296 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.131001 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.131207 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.131273 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.133739 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:43719: BlockingRecv error: recv got EOF from 127.2.182.3:43719 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:16.774345 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:16.768849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:38641)
0924 10:45:16.769195 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:16.769208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:16.769308 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:16.769608 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:16.769611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:16.769823 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:16.769921 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.769927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.770953 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.770956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.771499 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.771506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.771918 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.771921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.771981 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.772484 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:16.772495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:16.774232 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":140,"threads_started":1}
W20260924 10:45:16.774488 11976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:16.768934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:38641 (local address 127.2.182.62:32807)
0924 10:45:16.769204 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:16.769216 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:16.769264 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:45:16.769398 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:16.769401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:16.769454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:16.769574 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:16.770058 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.770857 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.771600 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.771827 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.771900 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.774358 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:38641: BlockingRecv error: recv got EOF from 127.2.182.1:38641 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:16 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:16.938724 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:16.933551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:52391)
0924 10:45:16.933853 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:16.933866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:16.933963 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:16.934177 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:16.934179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:16.934354 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:16.934448 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.934452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.935180 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.935182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.935777 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:16.935785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.936179 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.936181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:16.936254 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.936712 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:16.936723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:16.938535 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":118,"threads_started":1}
W20260924 10:45:16.938886 11976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:16.933601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:52391 (local address 127.2.182.62:32807)
0924 10:45:16.933751 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:16.933755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:16.933937 (+ 182us) server_negotiation.cc:407] Connection header received
0924 10:45:16.934029 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:16.934031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:16.934077 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:16.934144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:16.934541 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.935058 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.935874 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:16.936087 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:16.936162 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:16.938741 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:52391: BlockingRecv error: recv got EOF from 127.2.182.2:52391 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:17.140400 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:17.134482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:53051)
0924 10:45:17.134789 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:17.134802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:17.134899 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:17.135159 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:17.135162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:17.135365 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:17.135462 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.135468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.136531 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.136534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.137213 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.137223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.137704 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.137708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.137797 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.138269 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:17.138280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:17.140288 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":143,"threads_started":1}
W20260924 10:45:17.140542 11976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:17.134542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:53051 (local address 127.2.182.62:32807)
0924 10:45:17.134650 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:17.134654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:17.134871 (+ 217us) server_negotiation.cc:407] Connection header received
0924 10:45:17.134983 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:17.134985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:17.135030 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:17.135098 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:17.135619 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.136397 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.137341 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.137581 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.137689 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.140421 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:53051: BlockingRecv error: recv got EOF from 127.2.182.3:53051 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:17.780802 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:17.775214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:45579)
0924 10:45:17.775556 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:17.775572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:17.775683 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:17.775926 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:17.775928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:17.776121 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:17.776225 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.776231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.777002 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.777006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.777638 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.777649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.778173 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.778176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.778270 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.778870 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:17.778885 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:17.780680 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
W20260924 10:45:17.780969 11980 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:17.775335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:45579 (local address 127.2.182.62:32807)
0924 10:45:17.775597 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:17.775601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:17.775628 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:45:17.775754 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:17.775756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:17.775796 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:17.775871 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:17.776332 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.776862 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.777755 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.778042 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.778169 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.780804 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:45579: BlockingRecv error: recv got EOF from 127.2.182.1:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":64,"threads_started":1}
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:17 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:17.945192 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:17.939400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:58363)
0924 10:45:17.939680 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:17.939693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:17.939784 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:17.939985 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:17.939987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:17.940170 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:17.940262 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.940267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.941073 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.941077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.941644 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:17.941653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.942370 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.942373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:17.942445 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.943015 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:17.943026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:17.944994 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
W20260924 10:45:17.945300 11980 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:17.939526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:58363 (local address 127.2.182.62:32807)
0924 10:45:17.939683 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:17.939686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:17.939755 (+ 69us) server_negotiation.cc:407] Connection header received
0924 10:45:17.939848 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:17.939850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:17.939889 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:17.939954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:17.940359 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.940968 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.941754 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:17.942172 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:17.942372 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:17.945146 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:58363: BlockingRecv error: recv got EOF from 127.2.182.2:58363 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:18.146870 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:18.141213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:56209)
0924 10:45:18.141510 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:18.141527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:18.141679 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:18.141992 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:18.141995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:18.142230 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:18.142326 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.142332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.143128 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.143134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.143706 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.143713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.144177 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.144181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.144270 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.144856 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:18.144869 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:18.146755 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":75,"threads_started":1}
W20260924 10:45:18.147049 11980 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:18.141292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56209 (local address 127.2.182.62:32807)
0924 10:45:18.141428 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:18.141432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:18.141626 (+ 194us) server_negotiation.cc:407] Connection header received
0924 10:45:18.141806 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:18.141809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:18.141877 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:18.141967 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:18.142454 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.142998 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.143829 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.144062 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.144176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.146892 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:56209: BlockingRecv error: recv got EOF from 127.2.182.3:56209 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:18.791924 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:18.781601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:46317)
0924 10:45:18.781872 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:18.781892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:18.782017 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:18.782293 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:18.782296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:18.782473 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:18.782569 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.782574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.783301 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.783303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.783847 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.783854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.784248 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.784251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.784311 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.790016 (+ 5705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:18.790029 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:18.791813 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
W20260924 10:45:18.792130 11984 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:18.781651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:46317 (local address 127.2.182.62:32807)
0924 10:45:18.782001 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:18.782005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:18.782026 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:45:18.782137 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:18.782139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:18.782181 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:18.782247 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:18.782666 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.783204 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.783943 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.784156 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.784227 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.791940 (+ 7713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:46317: BlockingRecv error: recv got EOF from 127.2.182.1:46317 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":74,"threads_started":1}
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:18 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:18.951236 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:18.945911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:48151)
0924 10:45:18.946186 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:18.946200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:18.946296 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:18.946583 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:18.946585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:18.946767 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:18.946861 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.946867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.947578 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.947580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.948154 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:18.948161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.948633 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.948635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:18.948700 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.949199 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:18.949210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:18.951115 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
W20260924 10:45:18.951366 11984 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:18.946070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48151 (local address 127.2.182.62:32807)
0924 10:45:18.946218 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:18.946221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:18.946254 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:45:18.946416 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:18.946419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:18.946467 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:18.946541 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:18.946963 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.947480 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.948279 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:18.948522 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:18.948611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:18.951248 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:48151: BlockingRecv error: recv got EOF from 127.2.182.2:48151 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.3@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:19.153853 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:19.147688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.3:36515)
0924 10:45:19.147949 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:19.147965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:19.148092 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:19.148348 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:19.148350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:19.148530 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:19.148620 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.148625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.149401 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.149404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.149962 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.149969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.150388 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.150390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.150467 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.151050 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:19.151076 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:19.153648 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260924 10:45:19.154058 11984 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:19.147733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:36515 (local address 127.2.182.62:32807)
0924 10:45:19.147883 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:19.147887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:19.148017 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:45:19.148196 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:19.148199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:19.148247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:19.148313 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:19.148714 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.149311 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.150050 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.150285 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.150369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.153867 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.3:36515: BlockingRecv error: recv got EOF from 127.2.182.3:36515 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.1@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:19.798446 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:19.792831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.1:53085)
0924 10:45:19.793147 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:19.793161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:19.793247 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:19.793467 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:19.793469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:19.793660 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:19.793750 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.793756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.794484 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.794487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.795365 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.795378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.795971 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.795973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.796032 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.796576 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:19.796590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:19.798336 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":112,"threads_started":1}
W20260924 10:45:19.798593 11988 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:19.792983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:53085 (local address 127.2.182.62:32807)
0924 10:45:19.793220 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:19.793224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:19.793237 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:45:19.793316 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:19.793318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:19.793358 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:19.793432 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:19.793866 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.794405 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.795509 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.795859 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.795947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.798464 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.1:53085: BlockingRecv error: recv got EOF from 127.2.182.1:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":59,"threads_started":1}
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:45:19 dist-test-slave-6lwt krb5kdc[10929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.2@KRBTEST.COM for oryx/127.2.182.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:45:19.957821 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:19.952353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32807 (local address 127.2.182.2:40581)
0924 10:45:19.952615 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:19.952630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:19.952727 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:19.952999 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:19.953001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:19.953247 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:19.953378 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.953385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.954188 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.954191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.954850 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:19.954859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.955294 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.955296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:19.955356 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.955809 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:19.955820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:19.957701 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.62:32807: Server oryx/127.2.182.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260924 10:45:19.957984 11988 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:19.952372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:40581 (local address 127.2.182.62:32807)
0924 10:45:19.952518 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:19.952522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:19.952707 (+ 185us) server_negotiation.cc:407] Connection header received
0924 10:45:19.952794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:19.952796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:19.952840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:19.952979 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:19.953526 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.954103 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.954973 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:19.955200 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:19.955276 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:19.957830 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.2:40581: BlockingRecv error: recv got EOF from 127.2.182.2:40581 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:45:20.055721 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11512
I20260924 10:45:20.061794 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11665
I20260924 10:45:20.066996 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11797
I20260924 10:45:20.071902 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11442
2026-09-24T10:45:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17890 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T10:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:20Z Disabled control of system clock
I20260924 10:45:20.101989 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42067
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:45:20.209144 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:20.209371 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:20.209403 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:20.212911 11993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:20.212951 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:20.212965 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:20.212980 11993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:20.212993 11993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:20.217335 11993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:42067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:20.218252 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:20.219493 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:20.227586 12001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.227618 11998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.227782 11993 server_base.cc:1034] running on GCE node
W20260924 10:45:20.227669 11999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.228369 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:20.229485 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:20.230664 11993 hybrid_clock.cc:648] HybridClock initialized: now 1790246720230638 us; error 37 us; skew 500 ppm
I20260924 10:45:20.232349 11993 webserver.cc:460] Webserver started at http://127.2.182.62:40621/ using document root <none> and password file <none>
I20260924 10:45:20.232950 11993 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:20.233044 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:20.233263 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:20.235123 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "6acc92c57a004431b81450ea1dd01e9c"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "de1643b880a8197761ddc3875c40df3e"
server_key_iv: "7e428c95efb0ba8227eb09491ab96b19"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.235661 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "6acc92c57a004431b81450ea1dd01e9c"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "de1643b880a8197761ddc3875c40df3e"
server_key_iv: "7e428c95efb0ba8227eb09491ab96b19"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.239054 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:45:20.241333 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.242496 11993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:20.242635 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "6acc92c57a004431b81450ea1dd01e9c"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "de1643b880a8197761ddc3875c40df3e"
server_key_iv: "7e428c95efb0ba8227eb09491ab96b19"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.242740 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:20.256866 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:20.257467 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:20.257649 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:20.264227 11993 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:42067
I20260924 10:45:20.264235 12059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:42067 every 8 connection(s)
I20260924 10:45:20.265147 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:45:20.267131 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 11993
I20260924 10:45:20.267259 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:45:20.267635 2776 external_mini_cluster.cc:1428] Setting key f43c6992aa82335d4bf7e9ad766af514
I20260924 10:45:20.268257 12060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:20.274775 12060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c: Bootstrap starting.
I20260924 10:45:20.276413 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.269022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.2.182.62:42067)
0924 10:45:20.269560 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:20.269574 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:20.269607 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:45:20.270216 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:20.270231 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:20.270568 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:20.270821 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:20.271201 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.272112 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.273444 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.273879 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.273996 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.274824 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:20.274829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:20.274862 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:20.275226 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:20.275277 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:20.275295 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:20.275366 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:20.275523 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:45:20.275783 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":119,"threads_started":1}
I20260924 10:45:20.276964 12060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:20.277796 12060 log.cc:824] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:20.279472 12060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c: No bootstrap required, opened a new log
I20260924 10:45:20.282591 12060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } }
I20260924 10:45:20.282778 12060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:20.282864 12060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acc92c57a004431b81450ea1dd01e9c, State: Initialized, Role: FOLLOWER
I20260924 10:45:20.283354 12060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } }
I20260924 10:45:20.283528 12060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:45:20.283594 12060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:45:20.283687 12060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:20.284484 12060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } }
I20260924 10:45:20.284795 12060 leader_election.cc:302] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6acc92c57a004431b81450ea1dd01e9c; no voters:
I20260924 10:45:20.285121 12060 leader_election.cc:288] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:45:20.285215 12065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:20.285441 12065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [term 1 LEADER]: Becoming Leader. State: Replica: 6acc92c57a004431b81450ea1dd01e9c, State: Running, Role: LEADER
I20260924 10:45:20.285714 12065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } }
I20260924 10:45:20.286230 12060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:45:20.287465 12066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6acc92c57a004431b81450ea1dd01e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } } }
I20260924 10:45:20.287604 12066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:20.287825 12067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6acc92c57a004431b81450ea1dd01e9c. Latest consensus state: current_term: 1 leader_uuid: "6acc92c57a004431b81450ea1dd01e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acc92c57a004431b81450ea1dd01e9c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42067 } } }
I20260924 10:45:20.287940 12067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:20.288466 12074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:45:20.290886 12074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:45:20.295979 12074 catalog_manager.cc:1347] Generated new cluster ID: 8b6ad7d892fd4185847a3076f442b915
I20260924 10:45:20.296043 12074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:45:20.307188 12074 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:45:20.307983 12074 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:45:20.319427 12074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6acc92c57a004431b81450ea1dd01e9c: Generated new TSK 0
I20260924 10:45:20.320115 12074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:45:20.323523 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--builtin_ntp_servers=127.2.182.20:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 10:45:20.429221 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:20.429422 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:20.429453 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:20.433161 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:20.433199 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:20.433243 12084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:45:20.437777 12084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:20.438735 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:20.439970 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:20.449175 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.449185 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.449164 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.449625 12084 server_base.cc:1034] running on GCE node
I20260924 10:45:20.449995 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:20.450588 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:20.451777 12084 hybrid_clock.cc:648] HybridClock initialized: now 1790246720451768 us; error 31 us; skew 500 ppm
I20260924 10:45:20.453538 12084 webserver.cc:460] Webserver started at http://127.2.182.1:45539/ using document root <none> and password file <none>
I20260924 10:45:20.454082 12084 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:20.454169 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:20.454385 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:20.456244 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "26f7a89e97234db8ba85b92c7f88f2a6"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "912745d23bc42fabd7b363179f67a443"
server_key_iv: "9e979a9cd59e1a0c27dd687cbb49f0f6"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.456836 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "26f7a89e97234db8ba85b92c7f88f2a6"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "912745d23bc42fabd7b363179f67a443"
server_key_iv: "9e979a9cd59e1a0c27dd687cbb49f0f6"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.460198 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:45:20.462463 12098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.463560 12084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 10:45:20.463696 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "26f7a89e97234db8ba85b92c7f88f2a6"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "912745d23bc42fabd7b363179f67a443"
server_key_iv: "9e979a9cd59e1a0c27dd687cbb49f0f6"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.463804 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:20.478276 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:20.478906 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:20.479097 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:20.479667 12084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:20.480577 12084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:20.480647 12084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.480716 12084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:20.480767 12084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.489652 12084 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40847
I20260924 10:45:20.489727 12211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40847 every 8 connection(s)
I20260924 10:45:20.490546 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:45:20.499166 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.492321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:37705 (local address 127.2.182.62:42067)
0924 10:45:20.492464 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:20.492470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:20.493576 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 10:45:20.494909 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:20.494912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:20.494959 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:20.495043 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:20.495854 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.496361 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.497523 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.497733 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.497812 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.498769 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:20.498772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:20.498774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:20.498785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:20.498803 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:20.498810 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:20.498880 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:20.499014 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:45:20.499062 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:45:20.499547 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12084
I20260924 10:45:20.499635 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:45:20.499940 2776 external_mini_cluster.cc:1428] Setting key bb0d6ff811ee0581fd99493db54d8e69
I20260924 10:45:20.500006 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.492757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:42067 (local address 127.2.182.1:37705)
0924 10:45:20.493430 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:20.493468 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:20.494723 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:20.495202 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:20.495207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:20.495283 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:20.495737 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.495748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.496471 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.496473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.497411 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.497421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.497857 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.497859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.497995 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.498535 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:20.498536 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:20.498573 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:20.498629 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:20.498893 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:20.498898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:20.498902 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:45:20.498986 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:45:20.499185 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260924 10:45:20.501230 12212 heartbeater.cc:347] Connected to a master server at 127.2.182.62:42067
I20260924 10:45:20.501505 12212 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:20.501963 12212 heartbeater.cc:511] Master 127.2.182.62:42067 requested a full tablet report, sending...
I20260924 10:45:20.502015 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--builtin_ntp_servers=127.2.182.20:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:45:20.503539 12023 ts_manager.cc:194] Registered new tserver with Master: 26f7a89e97234db8ba85b92c7f88f2a6 (127.2.182.1:40847)
I20260924 10:45:20.505004 12023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:37705
W20260924 10:45:20.607601 12215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:20.607824 12215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:20.607859 12215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:20.611739 12215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:20.611781 12215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:20.611828 12215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:45:20.616199 12215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:20.617090 12215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:20.618309 12215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:20.627580 12220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.627592 12221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.627592 12223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.627874 12215 server_base.cc:1034] running on GCE node
I20260924 10:45:20.628266 12215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:20.628926 12215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:20.630120 12215 hybrid_clock.cc:648] HybridClock initialized: now 1790246720630107 us; error 24 us; skew 500 ppm
I20260924 10:45:20.631841 12215 webserver.cc:460] Webserver started at http://127.2.182.2:44537/ using document root <none> and password file <none>
I20260924 10:45:20.632386 12215 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:20.632472 12215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:20.632668 12215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:20.634601 12215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "5ac040b3309f4293a0ae3f812b6bdd05"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "53072e92181ca56f27f4666251f14205"
server_key_iv: "42da5d85da6b0e37a0d038a7cd5f4b9f"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.635160 12215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "5ac040b3309f4293a0ae3f812b6bdd05"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "53072e92181ca56f27f4666251f14205"
server_key_iv: "42da5d85da6b0e37a0d038a7cd5f4b9f"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.638682 12215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:45:20.640974 12229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.642076 12215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:20.642238 12215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "5ac040b3309f4293a0ae3f812b6bdd05"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "53072e92181ca56f27f4666251f14205"
server_key_iv: "42da5d85da6b0e37a0d038a7cd5f4b9f"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.642345 12215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:20.681576 12215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:20.682255 12215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:20.682443 12215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:20.683017 12215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:20.683926 12215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:20.683997 12215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.684062 12215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:20.684118 12215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.692739 12215 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:33445
I20260924 10:45:20.692780 12342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:33445 every 8 connection(s)
I20260924 10:45:20.693606 12215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:45:20.698637 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12215
I20260924 10:45:20.698768 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:45:20.699031 2776 external_mini_cluster.cc:1428] Setting key 792d04b832368f450dde4c487bdb682f
I20260924 10:45:20.701058 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--builtin_ntp_servers=127.2.182.20:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:45:20.702605 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.695285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:33217 (local address 127.2.182.62:42067)
0924 10:45:20.695410 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:20.695417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:20.696322 (+ 905us) server_negotiation.cc:407] Connection header received
0924 10:45:20.697726 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:20.697729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:20.697779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:20.697853 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:20.698698 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.699355 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.700715 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.700962 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.701077 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.702146 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:20.702149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:20.702152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:20.702170 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:20.702188 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:20.702196 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:20.702273 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:20.702420 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:45:20.702477 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:45:20.703433 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.695589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:42067 (local address 127.2.182.2:33217)
0924 10:45:20.696197 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:20.696235 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:20.697530 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:20.698008 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:20.698014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:20.698093 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:20.698537 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.698548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.699481 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.699484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.700584 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.700594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.701113 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.701117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.701275 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.701893 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:20.701895 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:20.701931 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:20.701989 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:20.702274 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:20.702285 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:20.702293 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:45:20.702420 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 10:45:20.702695 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260924 10:45:20.704493 12343 heartbeater.cc:347] Connected to a master server at 127.2.182.62:42067
I20260924 10:45:20.704738 12343 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:20.705252 12343 heartbeater.cc:511] Master 127.2.182.62:42067 requested a full tablet report, sending...
I20260924 10:45:20.706275 12023 ts_manager.cc:194] Registered new tserver with Master: 5ac040b3309f4293a0ae3f812b6bdd05 (127.2.182.2:33445)
I20260924 10:45:20.707116 12023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:33217
W20260924 10:45:20.809669 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:20.809906 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:20.809938 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:20.813721 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:20.813762 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:20.813810 12346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:20.818248 12346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:20.819108 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:20.820490 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:20.829654 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.829821 12346 server_base.cc:1034] running on GCE node
W20260924 10:45:20.829734 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:20.829660 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:20.830312 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:20.830853 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:20.832007 12346 hybrid_clock.cc:648] HybridClock initialized: now 1790246720831994 us; error 25 us; skew 500 ppm
I20260924 10:45:20.834026 12346 webserver.cc:460] Webserver started at http://127.2.182.3:40625/ using document root <none> and password file <none>
I20260924 10:45:20.834571 12346 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:20.834631 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:20.834853 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:20.836764 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "e0c2f7405187485c89cc56a3ee78bec0"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "588c200c4b3f07c64a39c6b092ce3910"
server_key_iv: "0496b566cb122b2dee4130ea815fb16e"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.837349 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "e0c2f7405187485c89cc56a3ee78bec0"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "588c200c4b3f07c64a39c6b092ce3910"
server_key_iv: "0496b566cb122b2dee4130ea815fb16e"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.840739 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:45:20.843150 12360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.844287 12346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:20.844434 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "e0c2f7405187485c89cc56a3ee78bec0"
format_stamp: "Formatted at 2026-09-24 10:45:20 on dist-test-slave-6lwt"
server_key: "588c200c4b3f07c64a39c6b092ce3910"
server_key_iv: "0496b566cb122b2dee4130ea815fb16e"
server_key_version: "encryptionkey@0"
I20260924 10:45:20.844555 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:20.859814 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:20.860422 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:20.860607 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:20.861161 12346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:20.862013 12346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:20.862082 12346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.862149 12346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:20.862191 12346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:20.871043 12346 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:36355
I20260924 10:45:20.871083 12473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:36355 every 8 connection(s)
I20260924 10:45:20.871927 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:45:20.877149 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12346
I20260924 10:45:20.877267 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:45:20.877573 2776 external_mini_cluster.cc:1428] Setting key 72a60a2661152dec6013ec9ab8e4133a
I20260924 10:45:20.881325 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.873817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:39197 (local address 127.2.182.62:42067)
0924 10:45:20.873956 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:20.873959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:20.874980 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 10:45:20.876321 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:20.876324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:20.876370 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:20.876446 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:20.877420 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.878152 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.879439 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.879790 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.879880 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.880857 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:20.880860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:20.880862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:20.880873 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:20.880937 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:20.880947 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:20.881016 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:20.881175 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 10:45:20.881223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:45:20.882153 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:20.874178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:42067 (local address 127.2.182.3:39197)
0924 10:45:20.874824 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:20.874861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:20.876133 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:20.876607 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:20.876613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:20.876690 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:20.877209 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.877222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.878283 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.878287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.879334 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:20.879344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:20.879940 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:20.879943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:20.880107 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:20.880638 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:20.880639 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:20.880675 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:20.880733 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:20.881030 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:20.881036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:20.881040 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:45:20.881160 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:45:20.881388 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260924 10:45:20.883222 12474 heartbeater.cc:347] Connected to a master server at 127.2.182.62:42067
I20260924 10:45:20.883443 12474 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:20.883934 12474 heartbeater.cc:511] Master 127.2.182.62:42067 requested a full tablet report, sending...
I20260924 10:45:20.884989 12023 ts_manager.cc:194] Registered new tserver with Master: e0c2f7405187485c89cc56a3ee78bec0 (127.2.182.3:36355)
I20260924 10:45:20.885540 12023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:39197
I20260924 10:45:20.891253 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:45:20.893911 12063 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:20.893132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.2.182.62:42067)
0924 10:45:20.893245 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:20.893248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:20.893451 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:45:20.893545 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:20.893547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:20.893584 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:20.893658 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:20.893823 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38932: BlockingRecv error: recv got EOF from 127.0.0.1:38932 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:45:20.894110 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12084
I20260924 10:45:20.899420 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12215
I20260924 10:45:20.904455 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12346
I20260924 10:45:20.909371 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 11993
2026-09-24T10:45:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (835 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T10:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:20Z Disabled control of system clock
I20260924 10:45:20.937276 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44623
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:44623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:45:21.045421 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:21.045666 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:21.045701 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:21.049805 12485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:21.049857 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:21.049873 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:21.049890 12485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:21.049904 12485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:21.054474 12485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:44623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:21.055353 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:21.056566 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:21.064472 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.064502 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.064720 12485 server_base.cc:1034] running on GCE node
W20260924 10:45:21.064745 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.065354 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:21.066351 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:21.067538 12485 hybrid_clock.cc:648] HybridClock initialized: now 1790246721067521 us; error 32 us; skew 500 ppm
I20260924 10:45:21.069684 12485 webserver.cc:460] Webserver started at http://127.2.182.62:46373/ using document root <none> and password file <none>
I20260924 10:45:21.070209 12485 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:21.070314 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:21.070508 12485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:21.072500 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "5cce10666a5347fbbeacb64bbc4419a5"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "4312b16a7d7a648e377c2dce180109e6"
server_key_iv: "b743d0d8c4564c94dca386a0826d1fac"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.073084 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "5cce10666a5347fbbeacb64bbc4419a5"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "4312b16a7d7a648e377c2dce180109e6"
server_key_iv: "b743d0d8c4564c94dca386a0826d1fac"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.076371 12485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:45:21.078656 12499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.079716 12485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:21.079846 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "5cce10666a5347fbbeacb64bbc4419a5"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "4312b16a7d7a648e377c2dce180109e6"
server_key_iv: "b743d0d8c4564c94dca386a0826d1fac"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.079950 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:21.084380 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:21.084983 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:21.085155 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:21.091383 12485 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:44623
I20260924 10:45:21.091383 12551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:44623 every 8 connection(s)
I20260924 10:45:21.092305 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:45:21.095153 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:21.100046 12552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5: Bootstrap starting.
I20260924 10:45:21.102209 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12485
I20260924 10:45:21.102231 12552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:21.102301 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:45:21.102559 2776 external_mini_cluster.cc:1428] Setting key 69389b4057504ea41d5607e4322b23cc
I20260924 10:45:21.103029 12552 log.cc:824] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:21.104991 12552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5: No bootstrap required, opened a new log
I20260924 10:45:21.107561 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.103656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.2.182.62:44623)
0924 10:45:21.104279 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:21.104294 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:21.104327 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:45:21.104951 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:21.104972 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:21.105276 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:21.105528 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:21.106358 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:21.106362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:21.106374 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:21.106383 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:21.106434 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:21.106449 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:21.106534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:21.106663 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 10:45:21.106916 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":144,"threads_started":1}
I20260924 10:45:21.108865 12552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } }
I20260924 10:45:21.109128 12552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:21.109180 12552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cce10666a5347fbbeacb64bbc4419a5, State: Initialized, Role: FOLLOWER
I20260924 10:45:21.109608 12552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } }
I20260924 10:45:21.109762 12552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:45:21.109812 12552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:45:21.109876 12552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:21.110594 12552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } }
I20260924 10:45:21.110838 12552 leader_election.cc:302] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cce10666a5347fbbeacb64bbc4419a5; no voters:
I20260924 10:45:21.111044 12552 leader_election.cc:288] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:45:21.111145 12557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:21.111378 12557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [term 1 LEADER]: Becoming Leader. State: Replica: 5cce10666a5347fbbeacb64bbc4419a5, State: Running, Role: LEADER
I20260924 10:45:21.111754 12557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } }
I20260924 10:45:21.112165 12552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:45:21.113286 12559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cce10666a5347fbbeacb64bbc4419a5. Latest consensus state: current_term: 1 leader_uuid: "5cce10666a5347fbbeacb64bbc4419a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } } }
I20260924 10:45:21.113286 12558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cce10666a5347fbbeacb64bbc4419a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce10666a5347fbbeacb64bbc4419a5" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44623 } } }
I20260924 10:45:21.113467 12559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:21.113503 12558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:21.113860 12566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:45:21.116389 12566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:45:21.121261 12566 catalog_manager.cc:1347] Generated new cluster ID: c1b7fba199b640c986e6486c4bfb25d8
I20260924 10:45:21.121335 12566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:45:21.133452 12566 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:45:21.133771 12566 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:45:21.138634 12566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cce10666a5347fbbeacb64bbc4419a5: Generated new TSK 0
I20260924 10:45:21.139262 12566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:45:21.148252 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--builtin_ntp_servers=127.2.182.20:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:45:21.258581 12576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:21.258812 12576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:21.258865 12576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:21.264376 12576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:21.264423 12576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:21.264503 12576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:45:21.268978 12576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:21.269907 12576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:21.271129 12576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:45:21.280014 12576 server_base.cc:1034] running on GCE node
W20260924 10:45:21.279990 12582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.279980 12584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.279980 12581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.280588 12576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:21.281235 12576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:21.282387 12576 hybrid_clock.cc:648] HybridClock initialized: now 1790246721282373 us; error 26 us; skew 500 ppm
I20260924 10:45:21.284334 12576 webserver.cc:460] Webserver started at http://127.2.182.1:45323/ using document root <none> and password file <none>
I20260924 10:45:21.284883 12576 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:21.285001 12576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:21.285224 12576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:21.287113 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "536d42940467419595824686e5a7ddf1"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "8d1894348986e9b13194add2a0a8e13c"
server_key_iv: "bbe1c59f919fd728e0b04f84f12c76f3"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.287693 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "536d42940467419595824686e5a7ddf1"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "8d1894348986e9b13194add2a0a8e13c"
server_key_iv: "bbe1c59f919fd728e0b04f84f12c76f3"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.291359 12576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:45:21.294250 12590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.295498 12576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:21.295634 12576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "536d42940467419595824686e5a7ddf1"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "8d1894348986e9b13194add2a0a8e13c"
server_key_iv: "bbe1c59f919fd728e0b04f84f12c76f3"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.295753 12576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:21.300362 12576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:21.301007 12576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:21.301186 12576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:21.301731 12576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:21.302578 12576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:21.302630 12576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.302695 12576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:21.302726 12576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.313174 12576 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:35919
I20260924 10:45:21.314200 12576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:45:21.314456 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12576
I20260924 10:45:21.314579 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:45:21.314913 2776 external_mini_cluster.cc:1428] Setting key a732be1ea3acc39b1bbe87f88a82cb16
I20260924 10:45:21.315055 12703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:35919 every 8 connection(s)
I20260924 10:45:21.317193 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--builtin_ntp_servers=127.2.182.20:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:45:21.322031 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.316768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:59147 (local address 127.2.182.62:44623)
0924 10:45:21.316882 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:21.316886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:21.318859 (+ 1973us) server_negotiation.cc:407] Connection header received
0924 10:45:21.320188 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:21.320192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:21.320231 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:21.320307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:21.321539 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:21.321542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:21.321546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:21.321558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:21.321580 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:21.321592 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:21.321690 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:21.321847 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:45:21.321891 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:45:21.322856 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.317348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44623 (local address 127.2.182.1:59147)
0924 10:45:21.318707 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:21.318769 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:21.319986 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:21.320466 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:21.320472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:21.320558 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:21.321293 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:21.321294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:21.321332 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:21.321396 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:21.321689 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:21.321700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:21.321707 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:45:21.321877 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 10:45:21.322116 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":115,"threads_started":1}
I20260924 10:45:21.324193 12704 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44623
I20260924 10:45:21.324430 12704 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:21.324808 12704 heartbeater.cc:511] Master 127.2.182.62:44623 requested a full tablet report, sending...
I20260924 10:45:21.326514 12516 ts_manager.cc:194] Registered new tserver with Master: 536d42940467419595824686e5a7ddf1 (127.2.182.1:35919)
W20260924 10:45:21.426182 12707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:21.426422 12707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:21.426460 12707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:21.430388 12707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:21.430444 12707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:21.430497 12707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:45:21.435534 12707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:21.436717 12707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:21.438114 12707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:21.447700 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.447726 12707 server_base.cc:1034] running on GCE node
W20260924 10:45:21.447692 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.447993 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.448395 12707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:21.449179 12707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:21.450371 12707 hybrid_clock.cc:648] HybridClock initialized: now 1790246721450354 us; error 29 us; skew 500 ppm
I20260924 10:45:21.452291 12707 webserver.cc:460] Webserver started at http://127.2.182.2:38943/ using document root <none> and password file <none>
I20260924 10:45:21.452986 12707 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:21.453080 12707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:21.453326 12707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:21.455446 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "c6688c16b1c0438181b4367d0ec62a5c"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "b9059ebfecfdd6f46b57ee16b7b82380"
server_key_iv: "6f27d42674041768e850cd353fb91acc"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.456020 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "c6688c16b1c0438181b4367d0ec62a5c"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "b9059ebfecfdd6f46b57ee16b7b82380"
server_key_iv: "6f27d42674041768e850cd353fb91acc"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.459717 12707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:45:21.462527 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.463837 12707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:45:21.463959 12707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "c6688c16b1c0438181b4367d0ec62a5c"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "b9059ebfecfdd6f46b57ee16b7b82380"
server_key_iv: "6f27d42674041768e850cd353fb91acc"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.464071 12707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:21.468429 12707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:21.469076 12707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:21.469255 12707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:21.469786 12707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:21.470669 12707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:21.470714 12707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.470778 12707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:21.470816 12707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.479238 12707 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:43341
I20260924 10:45:21.479259 12834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:43341 every 8 connection(s)
I20260924 10:45:21.480134 12707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:45:21.483364 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12707
I20260924 10:45:21.483490 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:45:21.483790 2776 external_mini_cluster.cc:1428] Setting key 932fb495c6d7fcde417dc43c9d9209aa
I20260924 10:45:21.486053 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--builtin_ntp_servers=127.2.182.20:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:45:21.486892 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.481959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:47153 (local address 127.2.182.62:44623)
0924 10:45:21.482100 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:21.482112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:21.483160 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 10:45:21.484974 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:21.484977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:21.485014 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:21.485087 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:21.486358 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:21.486364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:21.486370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:21.486386 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:21.486412 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:21.486426 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:21.486536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:21.486662 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 10:45:21.486708 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:45:21.487855 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.482362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44623 (local address 127.2.182.2:47153)
0924 10:45:21.482988 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:21.483026 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:21.484703 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:21.485313 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:21.485324 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:21.485439 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:21.486056 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:21.486058 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:21.486098 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:21.486197 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:21.486526 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:21.486532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:21.486536 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:45:21.486681 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:45:21.486869 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260924 10:45:21.489039 12835 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44623
I20260924 10:45:21.489281 12835 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:21.489741 12835 heartbeater.cc:511] Master 127.2.182.62:44623 requested a full tablet report, sending...
I20260924 10:45:21.490808 12516 ts_manager.cc:194] Registered new tserver with Master: c6688c16b1c0438181b4367d0ec62a5c (127.2.182.2:43341)
W20260924 10:45:21.602959 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:21.603256 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:21.603336 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:21.607362 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:21.607441 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:21.607529 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:21.612192 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:21.613401 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:21.614794 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:21.624353 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.624367 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:21.624367 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:21.624584 12838 server_base.cc:1034] running on GCE node
I20260924 10:45:21.625182 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:21.625757 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:21.626955 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790246721626914 us; error 57 us; skew 500 ppm
I20260924 10:45:21.628728 12838 webserver.cc:460] Webserver started at http://127.2.182.3:45767/ using document root <none> and password file <none>
I20260924 10:45:21.629375 12838 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:21.629436 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:21.629668 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:21.631798 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "9d920faa37a244c99886ec29ced04341"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "58a416b02bfcf91e6b7181cf13c9bb1f"
server_key_iv: "c3743db3e7e1d7c0bbe206f3ef268003"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.632349 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "9d920faa37a244c99886ec29ced04341"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "58a416b02bfcf91e6b7181cf13c9bb1f"
server_key_iv: "c3743db3e7e1d7c0bbe206f3ef268003"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.635932 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:45:21.638381 12852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.639583 12838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:45:21.639724 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "9d920faa37a244c99886ec29ced04341"
format_stamp: "Formatted at 2026-09-24 10:45:21 on dist-test-slave-6lwt"
server_key: "58a416b02bfcf91e6b7181cf13c9bb1f"
server_key_iv: "c3743db3e7e1d7c0bbe206f3ef268003"
server_key_version: "encryptionkey@0"
I20260924 10:45:21.639834 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:21.644268 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:21.644842 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:21.645037 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:21.645555 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:21.646417 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:21.646487 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.646553 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:21.646605 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:21.655323 12838 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:41373
I20260924 10:45:21.655342 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:41373 every 8 connection(s)
I20260924 10:45:21.656198 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:45:21.662089 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.657995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:58771 (local address 127.2.182.62:44623)
0924 10:45:21.658159 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:21.658165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:21.659175 (+ 1010us) server_negotiation.cc:407] Connection header received
0924 10:45:21.660406 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:21.660409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:21.660444 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:21.660519 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:21.661664 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:21.661667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:21.661669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:45:21.661680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:21.661700 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:45:21.661708 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:21.661783 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:21.661944 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 10:45:21.661981 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:45:21.662806 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12838
I20260924 10:45:21.662891 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:45:21.663009 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:21.658408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44623 (local address 127.2.182.3:58771)
0924 10:45:21.659034 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:21.659073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:21.660228 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:21.660685 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:21.660692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:21.660778 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:21.661415 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:45:21.661417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:21.661457 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:45:21.661522 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:21.661793 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:21.661801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:21.661806 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:45:21.661968 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 10:45:21.662180 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260924 10:45:21.663242 2776 external_mini_cluster.cc:1428] Setting key 728e3c9a01d6d334415babe539e39135
I20260924 10:45:21.664266 12966 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44623
I20260924 10:45:21.664533 12966 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:21.664992 12966 heartbeater.cc:511] Master 127.2.182.62:44623 requested a full tablet report, sending...
I20260924 10:45:21.665961 12516 ts_manager.cc:194] Registered new tserver with Master: 9d920faa37a244c99886ec29ced04341 (127.2.182.3:41373)
I20260924 10:45:21.677109 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:45:21.679927 12556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:45:21.679057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.2.182.62:44623)
0924 10:45:21.679224 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:21.679228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:21.679334 (+ 106us) server_negotiation.cc:407] Connection header received
0924 10:45:21.679430 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:21.679433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:21.679465 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:21.679533 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:21.679779 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45372: BlockingRecv error: recv got EOF from 127.0.0.1:45372 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:45:21.679997 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12576
I20260924 10:45:21.685498 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12707
I20260924 10:45:21.690706 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12838
I20260924 10:45:21.695986 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12485
2026-09-24T10:45:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (786 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:45:21 dist-test-slave-6lwt krb5kdc[12976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:45:21 dist-test-slave-6lwt krb5kdc[12976](info): set up 2 sockets
Sep 24 10:45:21 dist-test-slave-6lwt krb5kdc[12976](info): commencing operation
krb5kdc: starting...
W20260924 10:45:22.718406 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:45:22 dist-test-slave-6lwt krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:45:22.832273 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36451
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:22.937839 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:22.938112 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:22.938184 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:22.942194 12989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:45:22.942267 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:22.942313 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:22.942353 12989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:45:22.942389 12989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:45:22.947572 12989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:22.948544 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:22.949532 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:45:22.957809 12989 server_base.cc:1034] running on GCE node
W20260924 10:45:22.957764 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:22.957758 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:22.957764 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:22.958544 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:22.959522 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:22.960682 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790246722960651 us; error 44 us; skew 500 ppm
Sep 24 10:45:22 dist-test-slave-6lwt krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:22.963306 12989 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:45:22.964077 12989 webserver.cc:460] Webserver started at http://127.2.182.62:40143/ using document root <none> and password file <none>
I20260924 10:45:22.964556 12989 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:22.964620 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:22.964774 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:22.966729 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "50c72cdf22c8433d8f642b7880769577"
format_stamp: "Formatted at 2026-09-24 10:45:22 on dist-test-slave-6lwt"
server_key: "bf4335a61618113ab9138f592a7d1813"
server_key_iv: "5850c9adf252cac55c7ee305ce941e02"
server_key_version: "encryptionkey@0"
I20260924 10:45:22.967281 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "50c72cdf22c8433d8f642b7880769577"
format_stamp: "Formatted at 2026-09-24 10:45:22 on dist-test-slave-6lwt"
server_key: "bf4335a61618113ab9138f592a7d1813"
server_key_iv: "5850c9adf252cac55c7ee305ce941e02"
server_key_version: "encryptionkey@0"
I20260924 10:45:22.970774 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:45:22.972990 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:22.974138 12989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:22.974359 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "50c72cdf22c8433d8f642b7880769577"
format_stamp: "Formatted at 2026-09-24 10:45:22 on dist-test-slave-6lwt"
server_key: "bf4335a61618113ab9138f592a7d1813"
server_key_iv: "5850c9adf252cac55c7ee305ce941e02"
server_key_version: "encryptionkey@0"
I20260924 10:45:22.974457 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:22.991098 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:22.994204 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:22.994402 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:23.000972 12989 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36451
I20260924 10:45:23.001257 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36451 every 8 connection(s)
I20260924 10:45:23.001915 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:45:23.004526 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:23.007802 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 12989
I20260924 10:45:23.007936 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:45:23.008193 2776 external_mini_cluster.cc:1428] Setting key 95691f8c3c323b109339a57300573239
I20260924 10:45:23.010281 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: Bootstrap starting.
I20260924 10:45:23.012903 13057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:23.013780 13057 log.cc:824] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:23.015756 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: No bootstrap required, opened a new log
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:23.019840 13057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } }
I20260924 10:45:23.020133 13057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:23.020195 13057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c72cdf22c8433d8f642b7880769577, State: Initialized, Role: FOLLOWER
I20260924 10:45:23.020644 13057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } }
I20260924 10:45:23.020817 13057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:45:23.020912 13057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:45:23.021044 13057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:23.022112 13057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } }
I20260924 10:45:23.022434 13057 leader_election.cc:302] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c72cdf22c8433d8f642b7880769577; no voters:
I20260924 10:45:23.022732 13057 leader_election.cc:288] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:45:23.022791 13062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:23.022917 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.009575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.2.182.62:36451)
0924 10:45:23.009970 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.009987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.010028 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:45:23.010825 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.010848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.011237 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.011617 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:23.012247 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.013827 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.014647 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.015302 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.015454 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.018084 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:23.018089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:23.018100 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:23.018125 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:23.019839 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.020322 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.020328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.020334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.020487 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.020821 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.020825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.020827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.021350 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:23.021507 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.021794 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.022078 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260924 10:45:23.022975 13062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [term 1 LEADER]: Becoming Leader. State: Replica: 50c72cdf22c8433d8f642b7880769577, State: Running, Role: LEADER
I20260924 10:45:23.023249 13062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } }
I20260924 10:45:23.023904 13057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:45:23.024340 13063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c72cdf22c8433d8f642b7880769577. Latest consensus state: current_term: 1 leader_uuid: "50c72cdf22c8433d8f642b7880769577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } } }
I20260924 10:45:23.024348 13062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c72cdf22c8433d8f642b7880769577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c72cdf22c8433d8f642b7880769577" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36451 } } }
I20260924 10:45:23.024467 13063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:23.024474 13062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577 [sys.catalog]: This master's current role is: LEADER
I20260924 10:45:23.025132 13066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:45:23.027680 13066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:45:23.034284 13066 catalog_manager.cc:1347] Generated new cluster ID: 131811814ec44c3da323de70e8cce4ce
I20260924 10:45:23.034353 13066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:45:23.043186 13066 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:45:23.044032 13066 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:45:23.052091 13066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: Generated new TSK 0
I20260924 10:45:23.052767 13066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:45:23.083550 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:23.231998 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:23.232254 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:23.232293 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:23.236379 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:23.236431 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:23.236522 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:45:23.241605 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:23.242619 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:23.243662 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:23.252908 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:23.252928 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:23.252924 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:23.253155 13083 server_base.cc:1034] running on GCE node
I20260924 10:45:23.253638 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:23.254254 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:23.255440 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790246723255421 us; error 37 us; skew 500 ppm
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:23.258119 13083 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:45:23.258939 13083 webserver.cc:460] Webserver started at http://127.2.182.1:39219/ using document root <none> and password file <none>
I20260924 10:45:23.259478 13083 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:23.259569 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:23.259802 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:23.261822 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "aa8fc4efaa794c25bff26338f1d5012f"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "6726a06fb11197e6ae0f5bec8ff49dd4"
server_key_iv: "d8054b4418606609f2fe8548218848df"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.262365 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "aa8fc4efaa794c25bff26338f1d5012f"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "6726a06fb11197e6ae0f5bec8ff49dd4"
server_key_iv: "d8054b4418606609f2fe8548218848df"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.265990 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:45:23.268249 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.269456 13083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:23.269588 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "aa8fc4efaa794c25bff26338f1d5012f"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "6726a06fb11197e6ae0f5bec8ff49dd4"
server_key_iv: "d8054b4418606609f2fe8548218848df"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.269716 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:23.280499 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:23.283213 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:23.283399 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:23.283941 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:23.284703 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:23.284781 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.284848 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:23.284888 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.293308 13083 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:42761
I20260924 10:45:23.293347 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:42761 every 8 connection(s)
I20260924 10:45:23.294184 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:23.303362 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 13083
I20260924 10:45:23.303500 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:45:23.303810 2776 external_mini_cluster.cc:1428] Setting key 4d0c8a459b3bbdcc842571c6a5deb7fe
I20260924 10:45:23.307684 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.295910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:41141 (local address 127.2.182.62:36451)
0924 10:45:23.296056 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.296060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.296924 (+ 864us) server_negotiation.cc:407] Connection header received
0924 10:45:23.298250 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.298252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.298318 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.298391 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:23.299533 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.300068 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.301205 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.301424 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.301503 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.303908 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:23.303919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:23.303922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:23.303943 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:23.305414 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.305921 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.305926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.305928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.306008 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.306345 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.306350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.306352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.306526 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:23.306633 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.307279 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.307453 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:45:23.308508 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.296310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36451 (local address 127.2.182.1:41141)
0924 10:45:23.296753 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.296795 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.298059 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.298552 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.298558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.298929 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:23.299407 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.299419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.300175 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.300177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.301104 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.301113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.301548 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.301550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.301697 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.302231 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:23.302246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:23.303748 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:23.305536 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.305542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.305555 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.305817 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.306107 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.306110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.306112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.306249 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.306640 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:23.306648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:23.306994 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.307194 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.307511 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260924 10:45:23.309772 13212 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36451
I20260924 10:45:23.310014 13212 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:23.310479 13212 heartbeater.cc:511] Master 127.2.182.62:36451 requested a full tablet report, sending...
I20260924 10:45:23.311966 13021 ts_manager.cc:194] Registered new tserver with Master: aa8fc4efaa794c25bff26338f1d5012f (127.2.182.1:42761)
I20260924 10:45:23.313153 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:41141
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:45:23.331087 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:23.436957 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:23.437188 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:23.437227 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:23.441155 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:23.441198 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:23.441286 13217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:45:23.446348 13217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:23.447274 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:23.448232 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:23.457180 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:23.457227 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:23.457252 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:23.457643 13217 server_base.cc:1034] running on GCE node
I20260924 10:45:23.458010 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:23.458629 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:23.459803 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790246723459781 us; error 50 us; skew 500 ppm
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:23.462486 13217 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:45:23.463325 13217 webserver.cc:460] Webserver started at http://127.2.182.2:45031/ using document root <none> and password file <none>
I20260924 10:45:23.463868 13217 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:23.463958 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:23.464180 13217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:23.466171 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "f528704b4e454dedbf53a9cf480ab776"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "0b514b282897cc5c4e0be275720d56b4"
server_key_iv: "998892312874f6986841789c58e9ce6e"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.466714 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "f528704b4e454dedbf53a9cf480ab776"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "0b514b282897cc5c4e0be275720d56b4"
server_key_iv: "998892312874f6986841789c58e9ce6e"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.470216 13217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:45:23.472467 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.473690 13217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:45:23.473778 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "f528704b4e454dedbf53a9cf480ab776"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "0b514b282897cc5c4e0be275720d56b4"
server_key_iv: "998892312874f6986841789c58e9ce6e"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.473852 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:23.483768 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:23.486789 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:23.486930 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:23.487399 13217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:23.488145 13217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:23.488189 13217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.488226 13217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:23.488242 13217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.496946 13217 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:45297
I20260924 10:45:23.496946 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:45297 every 8 connection(s)
I20260924 10:45:23.497816 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:23.506613 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 13217
I20260924 10:45:23.506733 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:45:23.507017 2776 external_mini_cluster.cc:1428] Setting key 217b610202bde6766421c85f58277c9e
I20260924 10:45:23.511013 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.499504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:39317 (local address 127.2.182.62:36451)
0924 10:45:23.499651 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.499655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.500545 (+ 890us) server_negotiation.cc:407] Connection header received
0924 10:45:23.501846 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.501848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.501910 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.501983 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:23.503008 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.503548 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.504600 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.504808 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.504879 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.507241 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:23.507244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:23.507246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:23.507267 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:23.508994 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.509446 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.509452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.509456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.509546 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.509861 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.509867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.509871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.510125 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:23.510234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.510809 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.510901 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:45:23.512096 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.499934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36451 (local address 127.2.182.2:39317)
0924 10:45:23.500399 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.500438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.501672 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.502141 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.502147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.502455 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:23.502892 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.502904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.503650 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.503652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.504503 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.504510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.504972 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.504976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.505113 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.505658 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:23.505674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:23.507047 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:23.509100 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.509104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.509122 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.509350 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.509645 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.509647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.509648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.509774 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.510245 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:23.510254 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:23.510564 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.510769 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.511086 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260924 10:45:23.513191 13346 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36451
I20260924 10:45:23.513417 13346 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:23.513870 13346 heartbeater.cc:511] Master 127.2.182.62:36451 requested a full tablet report, sending...
I20260924 10:45:23.514937 13021 ts_manager.cc:194] Registered new tserver with Master: f528704b4e454dedbf53a9cf480ab776 (127.2.182.2:45297)
I20260924 10:45:23.515524 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:39317
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:45:23.533437 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:45:23.639739 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:45:23.639963 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:45:23.640002 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:45:23.643889 13351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:45:23.643937 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:45:23.643992 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:45:23.649013 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:45:23.649942 13351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:45:23.650903 13351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:45:23.660065 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:45:23.660076 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:23.660120 13351 server_base.cc:1034] running on GCE node
W20260924 10:45:23.660037 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:45:23.660586 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:45:23.661203 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:45:23.662349 13351 hybrid_clock.cc:648] HybridClock initialized: now 1790246723662319 us; error 42 us; skew 500 ppm
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:45:23.665058 13351 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:45:23.665823 13351 webserver.cc:460] Webserver started at http://127.2.182.3:38473/ using document root <none> and password file <none>
I20260924 10:45:23.666347 13351 fs_manager.cc:359] Metadata directory not provided
I20260924 10:45:23.666404 13351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:45:23.666618 13351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:45:23.668519 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "6af0ed1931e449dca2c57519de49001a"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "1eb41b3a19880f927f213b67958733af"
server_key_iv: "19f7c2c67562cfefe615cbe183b0052b"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.669124 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "6af0ed1931e449dca2c57519de49001a"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "1eb41b3a19880f927f213b67958733af"
server_key_iv: "19f7c2c67562cfefe615cbe183b0052b"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.672446 13351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:45:23.674695 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.675779 13351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:45:23.675905 13351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "6af0ed1931e449dca2c57519de49001a"
format_stamp: "Formatted at 2026-09-24 10:45:23 on dist-test-slave-6lwt"
server_key: "1eb41b3a19880f927f213b67958733af"
server_key_iv: "19f7c2c67562cfefe615cbe183b0052b"
server_key_version: "encryptionkey@0"
I20260924 10:45:23.676005 13351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:45:23.695797 13351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:45:23.698930 13351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:45:23.699131 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:45:23.699685 13351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:45:23.700455 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:45:23.700536 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.700594 13351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:45:23.700651 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:45:23.708971 13351 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:36719
I20260924 10:45:23.709007 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:36719 every 8 connection(s)
I20260924 10:45:23.709805 13351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:45:23 dist-test-slave-6lwt krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:45:23.719280 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 13351
I20260924 10:45:23.719403 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:45:23.719695 2776 external_mini_cluster.cc:1428] Setting key 349e311033a225b8550b114dbfad1985
I20260924 10:45:23.722371 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.711474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:39221 (local address 127.2.182.62:36451)
0924 10:45:23.711648 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.711652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.712391 (+ 739us) server_negotiation.cc:407] Connection header received
0924 10:45:23.713691 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.713693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.713739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.713805 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:23.714837 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.715348 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.716427 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.716653 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.716726 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.718932 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:23.718935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:23.718936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:23.718958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:23.720497 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.720972 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.720975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.720977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.721058 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.721372 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.721375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.721377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.721540 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:23.721640 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.722077 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.722203 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:45:23.723048 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.711823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36451 (local address 127.2.182.3:39221)
0924 10:45:23.712244 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.712281 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.713486 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.713952 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.713958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.714273 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:45:23.714714 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.714727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.715445 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.715447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.716331 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.716340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.716769 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.716771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.716917 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.717384 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:45:23.717399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:45:23.718777 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:45:23.720582 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.720588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.720600 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.720835 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.721145 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:45:23.721149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:45:23.721150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:45:23.721277 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:45:23.721650 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:45:23.721657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:45:23.721864 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.722025 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.722328 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260924 10:45:23.724082 13480 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36451
I20260924 10:45:23.724313 13480 heartbeater.cc:464] Registering TS with master...
I20260924 10:45:23.724795 13480 heartbeater.cc:511] Master 127.2.182.62:36451 requested a full tablet report, sending...
I20260924 10:45:23.725751 13021 ts_manager.cc:194] Registered new tserver with Master: 6af0ed1931e449dca2c57519de49001a (127.2.182.3:36719)
I20260924 10:45:23.726354 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:39221
I20260924 10:45:23.733035 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:45:23.741503 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.734776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.2.182.62:36451)
0924 10:45:23.734903 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.734908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.735021 (+ 113us) server_negotiation.cc:407] Connection header received
0924 10:45:23.735112 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.735114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.735158 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.735216 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:45:23.735917 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.736400 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.737295 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.737494 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.737568 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.738748 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:45:23.738751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:45:23.738753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:45:23.738772 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:45:23.740121 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.740452 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.740454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.740455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.740522 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:45:23.740728 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:45:23.740730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:45:23.740731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:45:23.740972 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:45:23.741070 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.741237 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.741320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:45:23.744064 13021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:45:23.747265 13021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:45:23.761842 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.755652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:36719 (local address 127.0.0.1:51944)
0924 10:45:23.756208 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.756258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.756380 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.757521 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.757524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.757541 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:23.757729 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.757737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.759227 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.759230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.760143 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.760153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761283 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.761404 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761445 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.761547 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.761693 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":134,"thread_start_us":89,"threads_started":1}
I20260924 10:45:23.761865 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.756885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.2.182.1:42761)
0924 10:45:23.757289 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.757299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.757316 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:45:23.757382 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.757386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.757536 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.757726 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:23.757888 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.759044 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.760559 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761217 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761430 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761568 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.761636 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.761722 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260924 10:45:23.762189 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.756459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.2.182.2:45297)
0924 10:45:23.756865 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.756877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.757056 (+ 179us) server_negotiation.cc:407] Connection header received
0924 10:45:23.757175 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.757180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.757349 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.757509 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:23.757831 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.758607 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.760401 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761198 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761773 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761905 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.761960 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.762058 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":124,"threads_started":1}
I20260924 10:45:23.761865 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.756320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.2.182.3:36719)
0924 10:45:23.757043 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.757056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.757078 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:45:23.757158 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.757165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.757369 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.757870 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:23.758005 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.759090 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.760270 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761136 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761275 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761430 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.761645 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.761729 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":157,"threads_started":1}
I20260924 10:45:23.762147 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.756043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:45297 (local address 127.0.0.1:55176)
0924 10:45:23.756950 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.756963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.757052 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.757498 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.757504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.757525 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:23.757703 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.757709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.758788 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.758794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.760276 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.760287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761338 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.761443 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761472 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.761529 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.761693 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":63,"spinlock_wait_cycles":2432,"thread_start_us":106,"threads_started":1}
I20260924 10:45:23.762372 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.755839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:42761 (local address 127.0.0.1:44224)
0924 10:45:23.756675 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.756688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.756766 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.757702 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.757705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.757721 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:23.757816 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.757821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.759215 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.759219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.760451 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.760459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.761354 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.761357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.761527 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.761546 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.761600 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.762241 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":669,"thread_start_us":111,"threads_started":1}
I20260924 10:45:23.765197 13414 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1473c0b92144098c0b8b7827edde8c (DEFAULT_TABLE table=test-table [id=d32dc941016d45f1a1fc0e4dce8abfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:23.765297 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1473c0b92144098c0b8b7827edde8c (DEFAULT_TABLE table=test-table [id=d32dc941016d45f1a1fc0e4dce8abfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:23.765298 13146 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1473c0b92144098c0b8b7827edde8c (DEFAULT_TABLE table=test-table [id=d32dc941016d45f1a1fc0e4dce8abfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:45:23.766264 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1473c0b92144098c0b8b7827edde8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:23.766275 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1473c0b92144098c0b8b7827edde8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:23.766275 13146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1473c0b92144098c0b8b7827edde8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:45:23.772548 13497 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Bootstrap starting.
I20260924 10:45:23.774034 13498 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Bootstrap starting.
I20260924 10:45:23.774431 13499 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Bootstrap starting.
I20260924 10:45:23.775523 13497 tablet_bootstrap.cc:654] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:23.776252 13497 log.cc:824] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:23.776515 13499 tablet_bootstrap.cc:654] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:23.777305 13499 log.cc:824] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:23.778566 13498 tablet_bootstrap.cc:654] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Neither blocks nor log segments found. Creating new log.
I20260924 10:45:23.778581 13497 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: No bootstrap required, opened a new log
I20260924 10:45:23.778777 13497 ts_tablet_manager.cc:1397] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:45:23.779426 13499 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: No bootstrap required, opened a new log
I20260924 10:45:23.779625 13499 ts_tablet_manager.cc:1397] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:45:23.779783 13498 log.cc:824] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:45:23.781493 13498 tablet_bootstrap.cc:492] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: No bootstrap required, opened a new log
I20260924 10:45:23.781693 13498 ts_tablet_manager.cc:1397] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 10:45:23.782826 13497 raft_consensus.cc:348] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.783090 13497 raft_consensus.cc:374] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:23.783211 13497 raft_consensus.cc:729] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0ed1931e449dca2c57519de49001a, State: Initialized, Role: FOLLOWER
I20260924 10:45:23.783399 13499 raft_consensus.cc:348] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.783629 13499 raft_consensus.cc:374] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:23.783722 13499 raft_consensus.cc:729] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f528704b4e454dedbf53a9cf480ab776, State: Initialized, Role: FOLLOWER
I20260924 10:45:23.783684 13497 consensus_queue.cc:260] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.784160 13499 consensus_queue.cc:260] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.784371 13480 heartbeater.cc:503] Master 127.2.182.62:36451 was elected leader, sending a full tablet report...
I20260924 10:45:23.784696 13497 ts_tablet_manager.cc:1428] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:45:23.785104 13499 ts_tablet_manager.cc:1428] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:45:23.784991 13498 raft_consensus.cc:348] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.785218 13498 raft_consensus.cc:374] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:45:23.785269 13498 raft_consensus.cc:729] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8fc4efaa794c25bff26338f1d5012f, State: Initialized, Role: FOLLOWER
I20260924 10:45:23.785398 13346 heartbeater.cc:503] Master 127.2.182.62:36451 was elected leader, sending a full tablet report...
I20260924 10:45:23.785663 13498 consensus_queue.cc:260] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.786330 13212 heartbeater.cc:503] Master 127.2.182.62:36451 was elected leader, sending a full tablet report...
I20260924 10:45:23.786849 13498 ts_tablet_manager.cc:1428] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260924 10:45:23.795298 13213 tablet.cc:2369] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:45:23.900837 13503 raft_consensus.cc:482] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:45:23.901062 13503 raft_consensus.cc:504] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.902143 13503 leader_election.cc:288] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa8fc4efaa794c25bff26338f1d5012f (127.2.182.1:42761), f528704b4e454dedbf53a9cf480ab776 (127.2.182.2:45297)
I20260924 10:45:23.906085 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.902371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:45297 (local address 127.2.182.3:46469)
0924 10:45:23.902499 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.902516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.902614 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.902925 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.902927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.902945 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:23.903138 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.903161 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.904039 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.904043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.904958 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.904967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.905629 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.905751 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.905790 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.905860 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.905921 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":68,"spinlock_wait_cycles":26880}
I20260924 10:45:23.906141 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.902481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:46469 (local address 127.2.182.2:45297)
0924 10:45:23.902600 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.902606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.902624 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:45:23.902763 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.902765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.902817 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.902904 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:23.903317 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.903924 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905068 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.905511 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905589 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.905636 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.905890 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.905979 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:45:23.906188 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.902452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:52461 (local address 127.2.182.1:42761)
0924 10:45:23.902571 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.902576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.903001 (+ 425us) server_negotiation.cc:407] Connection header received
0924 10:45:23.903139 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.903156 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.903233 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.903315 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:45:23.903580 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.904112 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905215 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.905626 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.905804 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.906001 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.906078 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:45:23.906246 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.902373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:42761 (local address 127.2.182.3:52461)
0924 10:45:23.902899 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.902913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:45:23.903027 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:45:23.903339 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:45:23.903342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:45:23.903360 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:45:23.903474 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.903480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.904223 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.904226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.905096 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:45:23.905106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.905752 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.905754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:45:23.905875 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.905899 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:45:23.905970 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:45:23.906040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":152,"spinlock_wait_cycles":768,"thread_start_us":134,"threads_started":1}
I20260924 10:45:23.906893 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1473c0b92144098c0b8b7827edde8c" candidate_uuid: "6af0ed1931e449dca2c57519de49001a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8fc4efaa794c25bff26338f1d5012f" is_pre_election: true
I20260924 10:45:23.907090 13300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1473c0b92144098c0b8b7827edde8c" candidate_uuid: "6af0ed1931e449dca2c57519de49001a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f528704b4e454dedbf53a9cf480ab776" is_pre_election: true
I20260924 10:45:23.907208 13166 raft_consensus.cc:2454] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0ed1931e449dca2c57519de49001a in term 0.
I20260924 10:45:23.907369 13300 raft_consensus.cc:2454] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0ed1931e449dca2c57519de49001a in term 0.
I20260924 10:45:23.907642 13368 leader_election.cc:302] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0ed1931e449dca2c57519de49001a, aa8fc4efaa794c25bff26338f1d5012f; no voters:
I20260924 10:45:23.907912 13503 raft_consensus.cc:2781] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:45:23.907991 13503 raft_consensus.cc:482] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:45:23.908034 13503 raft_consensus.cc:3037] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:23.909085 13503 raft_consensus.cc:504] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.909382 13503 leader_election.cc:288] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [CANDIDATE]: Term 1 election: Requested vote from peers aa8fc4efaa794c25bff26338f1d5012f (127.2.182.1:42761), f528704b4e454dedbf53a9cf480ab776 (127.2.182.2:45297)
I20260924 10:45:23.909662 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1473c0b92144098c0b8b7827edde8c" candidate_uuid: "6af0ed1931e449dca2c57519de49001a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8fc4efaa794c25bff26338f1d5012f"
I20260924 10:45:23.909724 13300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1473c0b92144098c0b8b7827edde8c" candidate_uuid: "6af0ed1931e449dca2c57519de49001a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f528704b4e454dedbf53a9cf480ab776"
I20260924 10:45:23.909791 13166 raft_consensus.cc:3037] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:23.909843 13300 raft_consensus.cc:3037] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:45:23.910686 13166 raft_consensus.cc:2454] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0ed1931e449dca2c57519de49001a in term 1.
I20260924 10:45:23.910749 13300 raft_consensus.cc:2454] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0ed1931e449dca2c57519de49001a in term 1.
I20260924 10:45:23.910934 13368 leader_election.cc:302] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0ed1931e449dca2c57519de49001a, aa8fc4efaa794c25bff26338f1d5012f; no voters:
I20260924 10:45:23.911072 13503 raft_consensus.cc:2781] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:45:23.911312 13503 raft_consensus.cc:686] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 1 LEADER]: Becoming Leader. State: Replica: 6af0ed1931e449dca2c57519de49001a, State: Running, Role: LEADER
I20260924 10:45:23.911604 13503 consensus_queue.cc:237] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } }
I20260924 10:45:23.914479 13020 catalog_manager.cc:5495] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0ed1931e449dca2c57519de49001a (127.2.182.3). New cstate: current_term: 1 leader_uuid: "6af0ed1931e449dca2c57519de49001a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0ed1931e449dca2c57519de49001a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 } health_report { overall_health: UNKNOWN } } }
I20260924 10:45:23.951437 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.948287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.2.182.3:36719)
0924 10:45:23.948408 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:45:23.948412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:45:23.948424 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:45:23.948556 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:45:23.948558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:45:23.948616 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:45:23.948701 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:45:23.948946 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.949398 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.950180 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:45:23.950369 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:45:23.950461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:45:23.950701 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:45:23.951088 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:45:23.951170 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:45:23.951227 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 10:45:23.951271 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:45:23.956972 13300 raft_consensus.cc:1261] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 1 FOLLOWER]: Refusing update from remote peer 6af0ed1931e449dca2c57519de49001a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:23.956916 13166 raft_consensus.cc:1261] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 1 FOLLOWER]: Refusing update from remote peer 6af0ed1931e449dca2c57519de49001a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:45:23.957690 13503 consensus_queue.cc:1035] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8fc4efaa794c25bff26338f1d5012f" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 42761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:45:23.957876 13507 consensus_queue.cc:1035] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f528704b4e454dedbf53a9cf480ab776" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:45:23.960672 13481 tablet.cc:2369] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:45:23.973829 13019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:23.973982 13019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:45:23.976294 13019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50c72cdf22c8433d8f642b7880769577: Sending DeleteTablet for 3 replicas of tablet 2b1473c0b92144098c0b8b7827edde8c
I20260924 10:45:23.976972 13414 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1473c0b92144098c0b8b7827edde8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260924 10:45:23.977049 13146 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1473c0b92144098c0b8b7827edde8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44224
I20260924 10:45:23.977445 13521 tablet_replica.cc:330] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: stopping tablet replica
I20260924 10:45:23.977445 13520 tablet_replica.cc:330] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f: stopping tablet replica
I20260924 10:45:23.977455 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1473c0b92144098c0b8b7827edde8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:45:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:55176
I20260924 10:45:23.977736 13520 raft_consensus.cc:2229] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:23.977736 13521 raft_consensus.cc:2229] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:45:23.978066 13520 raft_consensus.cc:2258] T 2b1473c0b92144098c0b8b7827edde8c P aa8fc4efaa794c25bff26338f1d5012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:23.978106 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 13083
I20260924 10:45:23.978052 13521 raft_consensus.cc:2258] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:23.979941 13521 ts_tablet_manager.cc:1905] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:45:23.983175 13521 ts_tablet_manager.cc:1918] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:45:23.983315 13521 log.cc:1196] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/2b1473c0b92144098c0b8b7827edde8c
I20260924 10:45:23.983465 13522 tablet_replica.cc:330] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: stopping tablet replica
I20260924 10:45:23.983742 13521 ts_tablet_manager.cc:1939] T 2b1473c0b92144098c0b8b7827edde8c P 6af0ed1931e449dca2c57519de49001a: Deleting consensus metadata
I20260924 10:45:23.983820 13522 raft_consensus.cc:2229] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:45:23.984140 13522 raft_consensus.cc:2258] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:45:23.984920 13005 catalog_manager.cc:4841] TS 6af0ed1931e449dca2c57519de49001a (127.2.182.3:36719): tablet 2b1473c0b92144098c0b8b7827edde8c (table test-table [id=d32dc941016d45f1a1fc0e4dce8abfa6]) successfully deleted
I20260924 10:45:23.985494 13522 ts_tablet_manager.cc:1905] T 2b1473c0b92144098c0b8b7827edde8c P f528704b4e454dedbf53a9cf480ab776: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:45:23.986430 13006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:42761 (error 108)
I20260924 10:45:23.987380 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.987061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:42761 (local address 127.0.0.1:44230)
0924 10:45:23.987196 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.987267 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:45:23.987480 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 13217
W20260924 10:45:23.987648 13006 catalog_manager.cc:4584] TS aa8fc4efaa794c25bff26338f1d5012f (127.2.182.1:42761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b1473c0b92144098c0b8b7827edde8c: Network error: Client connection negotiation failed: client connection to 127.2.182.1:42761: connect: Connection refused (error 111)
I20260924 10:45:23.993502 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:45:23.993263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:45297 (local address 127.0.0.1:55180)
0924 10:45:23.993386 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:45:23.993435 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 10:45:23.993506 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 13351
I20260924 10:45:23.999220 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 12989
2026-09-24T10:45:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2304 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:45:24 dist-test-slave-6lwt krb5kdc[13526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:45:24 dist-test-slave-6lwt krb5kdc[13526](info): set up 2 sockets
Sep 24 10:45:24 dist-test-slave-6lwt krb5kdc[13526](info): commencing operation
krb5kdc: starting...
W20260924 10:45:25.022364 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:45:25 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:45:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangeradmin_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangeradmin_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangerlookup_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangerlookup_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
I20260924 10:45:25.189191 2776 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwg6lLW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/postgres -l logfile start
2026-09-24 10:45:27.584 UTC [13556] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:45:27.584 UTC [13556] LOG: listening on IPv4 address "127.2.182.20", port 50475
2026-09-24 10:45:27.587 UTC [13556] LOG: listening on Unix socket "/tmp/.s.PGSQL.50475"
2026-09-24 10:45:27.608 UTC [13559] LOG: database system was shut down at 2026-09-24 10:45:26 UTC
2026-09-24 10:45:27.611 UTC [13556] LOG: database system is ready to accept connections
I20260924 10:45:28.554008 2776 mini_postgres.cc:96] Postgres bound to 50475
2026-09-24 10:45:28.560 UTC [13567] FATAL: database "slave" does not exist
127.2.182.20:50475 - accepting connections
I20260924 10:45:28.561116 2776 mini_ranger.cc:161] Starting Ranger...
I20260924 10:45:28.575508 2776 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:45:29.025574 2776 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:45:29.025678 2776 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin
2026-09-24 10:45:29,305 [I] DB FLAVOR :POSTGRES
2026-09-24 10:45:29,306 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:45:29,306 [I] Checking connection..
2026-09-24 10:45:29,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:45:29,543 [I] Checking connection passed.
2026-09-24 10:45:29,543 [I] --------- Verifying version history table ---------
2026-09-24 10:45:29,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:45:29,793 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:45:29,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:45:30,041 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:45:30,041 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:45:30,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:45:30.269 UTC [13631] WARNING: there is no transaction in progress
2026-09-24 10:45:30,275 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:45:30,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:45:30,528 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:45:30,528 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:45:30,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:45:30,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:45:31,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6lwt.c.gcp-upstream.internal','N') ;"
2026-09-24 10:45:31,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:45:31,470 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:45:31,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:45:31,703 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:45:31,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:45:31,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:45:32,165 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:45:32,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:45:33.063 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.078 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.093 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.107 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.122 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.139 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.197 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.204 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.219 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.229 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.239 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.253 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.262 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.268 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.274 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33.281 UTC [13739] WARNING: there is no transaction in progress
2026-09-24 10:45:33,917 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:45:33,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:45:34,156 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:45:34,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:45:34,411 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:45:34,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:45:34,665 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:45:34,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:45:34,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:45:35,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6lwt.c.gcp-upstream.internal';"
2026-09-24 10:45:35,350 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:45:35,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:45:35,603 [I] DB_PATCHES have already been applied
I20260924 10:45:35.609577 2776 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwg6lLW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:45:35.609644 2776 mini_ranger.cc:193] Using host: 127.2.182.20
I20260924 10:45:35.612996 2776 mini_ranger.cc:227] Ranger admin URL: http://127.2.182.20:44517
Sep 24, 2026 10:45:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:45:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:45:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:45:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:45:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:45:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:45:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:45:36.713493 2776 mini_ranger.cc:160] Time spent starting Ranger: real 8.152s user 0.003s sys 0.007s
Sep 24, 2026 10:45:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:45:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 10:45:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:45:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:45:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:45:56.447003 2776 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangerkms_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/rangerkms_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.182.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:45:56 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246756, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:45:56.530303 2776 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:45:56.539561 2776 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:45:57.381212 2776 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:45:57.381312 2776 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms
I20260924 10:45:57.381429 2776 mini_ranger_kms.cc:227] Using postgres at 127.2.182.20:50475
2026-09-24 10:45:57,560 [I] DB FLAVOR :POSTGRES
2026-09-24 10:45:57,560 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:45:57,560 [I] Checking connection
2026-09-24 10:45:57,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:45:57,815 [I] connection success
2026-09-24 10:45:57,815 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:45:57,815 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:45:57,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:45:58,033 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:45:58,033 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:45:58,033 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:45:58,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:50475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:45:58,281 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:45:58.730690 2776 mini_ranger_kms.cc:310] Created kms service
I20260924 10:45:58.932964 2776 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:45:58.975991 2776 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:45:58 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246756, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:45:59.346863 2776 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:45:59.347009 2776 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:45:59.347033 2776 mini_ranger_kms.cc:240] Using host: 127.2.182.20
I20260924 10:45:59.350584 2776 mini_ranger_kms.cc:276] Ranger KMS PID: 13919
I20260924 10:45:59.350725 2776 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.182.20:37985
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 10:46:01 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:01.566339 2776 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.036s user 0.000s sys 0.162s
I20260924 10:46:01.566480 2776 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:46:01.566591 2776 mini_ranger_kms.cc:401] 127.2.182.20:37985/kms/v1/keys
Sep 24 10:46:01 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
Sep 24 10:46:02 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
Sep 24 10:46:04 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:46:04.460691 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:33955
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:33955
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:37985/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:04.567811 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:04.568069 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:04.568137 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:04.572029 13970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:04.572106 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:04.572144 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:04.572192 13970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:04.572230 13970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:04.577185 13970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.182.20:37985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:33955
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:04.578092 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:04.578980 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:04.587548 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:04.587584 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:04.587520 13978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:04.587677 13970 server_base.cc:1034] running on GCE node
I20260924 10:46:04.588147 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:04.589218 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:04.590405 13970 hybrid_clock.cc:648] HybridClock initialized: now 1790246764590382 us; error 36 us; skew 500 ppm
Sep 24 10:46:04 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246764, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:04.596436 13970 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:46:04.597316 13970 webserver.cc:460] Webserver started at http://127.2.182.62:40867/ using document root <none> and password file <none>
I20260924 10:46:04.597811 13970 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:04.597884 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:04.598095 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:04 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246764, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:04.651151 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "bd59648ea4bc43c3849b9c5af5c876ba"
format_stamp: "Formatted at 2026-09-24 10:46:04 on dist-test-slave-6lwt"
server_key: "3daa35051df7b45689109ef00ae9a649"
server_key_iv: "7436650503fed8f954ea5f57e5afc6a8"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:04.651885 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "bd59648ea4bc43c3849b9c5af5c876ba"
format_stamp: "Formatted at 2026-09-24 10:46:04 on dist-test-slave-6lwt"
server_key: "3daa35051df7b45689109ef00ae9a649"
server_key_iv: "7436650503fed8f954ea5f57e5afc6a8"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:04.656863 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:46:04.674297 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:04.675707 13970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:46:04.675855 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "bd59648ea4bc43c3849b9c5af5c876ba"
format_stamp: "Formatted at 2026-09-24 10:46:04 on dist-test-slave-6lwt"
server_key: "3daa35051df7b45689109ef00ae9a649"
server_key_iv: "7436650503fed8f954ea5f57e5afc6a8"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:04.675987 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:04.689865 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:04.692692 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:04.692883 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:04.700110 13970 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:33955
I20260924 10:46:04.700170 14037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:33955 every 8 connection(s)
I20260924 10:46:04.701175 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:46:04.703895 14038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:04.707736 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 13970
I20260924 10:46:04.707861 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:46:04.708125 2776 external_mini_cluster.cc:1428] Setting key 17801f2f37dd9e7ca33ab4da20c38c63
I20260924 10:46:04.709444 14038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: Bootstrap starting.
I20260924 10:46:04.712070 14038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:04.713322 14038 log.cc:824] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:04.715191 14038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: No bootstrap required, opened a new log
Sep 24 10:46:04 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:04.719455 14038 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } }
I20260924 10:46:04.719728 14038 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:04.719899 14038 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd59648ea4bc43c3849b9c5af5c876ba, State: Initialized, Role: FOLLOWER
I20260924 10:46:04.720435 14038 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } }
I20260924 10:46:04.720614 14038 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:46:04.720712 14038 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:46:04.720860 14038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:04.721889 14038 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } }
I20260924 10:46:04.722302 14038 leader_election.cc:302] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd59648ea4bc43c3849b9c5af5c876ba; no voters:
I20260924 10:46:04.722692 14038 leader_election.cc:288] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:46:04.722755 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:04.723027 14043 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [term 1 LEADER]: Becoming Leader. State: Replica: bd59648ea4bc43c3849b9c5af5c876ba, State: Running, Role: LEADER
I20260924 10:46:04.723377 14043 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } }
I20260924 10:46:04.724102 14038 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:04.724624 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } } }
I20260924 10:46:04.724808 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:04.725095 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd59648ea4bc43c3849b9c5af5c876ba. Latest consensus state: current_term: 1 leader_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd59648ea4bc43c3849b9c5af5c876ba" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33955 } } }
I20260924 10:46:04.725246 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:04.725337 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:04.709532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.2.182.62:33955)
0924 10:46:04.709940 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:04.709958 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:04.709997 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:46:04.710838 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:04.710860 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:04.711240 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:04.711581 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:04.712441 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:04.714189 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:04.715383 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:04.716020 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:04.716174 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:04.719261 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:04.719268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:04.719285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:04.719314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:04.721351 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:04.721920 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:04.721926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:04.721932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:04.722062 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:04.722381 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:04.722384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:04.722386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:04.722881 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:04.723843 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:04.724060 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 10:46:04.724372 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260924 10:46:04.725565 14050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:46:04.729526 14050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:46:04.729959 14038 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:46:04.740811 14050 catalog_manager.cc:1347] Generated new cluster ID: dd11a70d2e654854a3725cac00d30ec4
I20260924 10:46:04.740970 14050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:46:04.775444 14050 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:46:04.776863 14050 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:46:04.783718 14050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: Generated new TSK 0
I20260924 10:46:04.784555 14050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:46:05.472291 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:37985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:05.623363 14083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:05.623670 14083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:05.623746 14083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:05.628181 14083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:05.628271 14083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:05.628371 14083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:46:05.633591 14083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.182.20:37985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:05.634655 14083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:05.635654 14083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:05.645319 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:05.645888 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:05.646126 14083 server_base.cc:1034] running on GCE node
W20260924 10:46:05.646620 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:05.647131 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:05.647933 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:05.649080 14083 hybrid_clock.cc:648] HybridClock initialized: now 1790246765649048 us; error 48 us; skew 500 ppm
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:05.653026 14083 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:46:05.653967 14083 webserver.cc:460] Webserver started at http://127.2.182.1:39741/ using document root <none> and password file <none>
I20260924 10:46:05.654517 14083 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:05.654605 14083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:05.654811 14083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:05.672279 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "beefdb84b555430c89e36d8f76f83377"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "f3706944979373c4db06e6e253d81b34"
server_key_iv: "d7eead8f5e6b4cd95fc2c87badd3dbcd"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:05.674472 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "beefdb84b555430c89e36d8f76f83377"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "f3706944979373c4db06e6e253d81b34"
server_key_iv: "d7eead8f5e6b4cd95fc2c87badd3dbcd"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:05.679553 14083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:46:05.698563 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:05.700722 14083 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 10:46:05.700949 14083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "beefdb84b555430c89e36d8f76f83377"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "f3706944979373c4db06e6e253d81b34"
server_key_iv: "d7eead8f5e6b4cd95fc2c87badd3dbcd"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:05.701175 14083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:05.736829 14083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:05.740427 14083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:05.740633 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:05.741387 14083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:05.742388 14083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:05.742486 14083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:05.742571 14083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:05.742624 14083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:05.753861 14083 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36781
I20260924 10:46:05.753978 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:36781 every 8 connection(s)
I20260924 10:46:05.754879 14083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:46:05.760941 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 14083
I20260924 10:46:05.761063 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:46:05.761370 2776 external_mini_cluster.cc:1428] Setting key d95a436ebdb959eef12cccc879f2311e
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:05.781960 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:05.757473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:33955 (local address 127.2.182.1:39233)
0924 10:46:05.765480 (+ 8007us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:05.765543 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:05.766635 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:05.767558 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:05.767568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:05.767935 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:05.768640 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:05.768662 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:05.769885 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:05.769888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:05.771112 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:05.771122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:05.771809 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:05.771814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:05.771979 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:05.772615 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:05.772632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:05.776329 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:05.778487 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:05.778493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:05.778511 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:05.778859 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:05.779215 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:05.779218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:05.779220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:05.779413 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:05.779909 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:05.779918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:05.780282 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 10:46:05.780474 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 10:46:05.780835 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7771,"thread_start_us":131,"threads_started":1}
I20260924 10:46:05.783320 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:05.757108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:39233 (local address 127.2.182.62:33955)
0924 10:46:05.765149 (+ 8041us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:05.765157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:05.766982 (+ 1825us) server_negotiation.cc:407] Connection header received
0924 10:46:05.767049 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:05.767052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:05.767104 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:05.767200 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:05.768824 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:05.769734 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:05.771263 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:05.771625 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:05.771764 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:05.776505 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:05.776510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:05.776512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:05.776540 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:05.778344 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:05.778992 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:05.778995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:05.778997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:05.779085 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:05.779539 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:05.779542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:05.779545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:05.779784 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:05.782925 (+ 3141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:05.783003 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:46:05.783130 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7948,"thread_start_us":94,"threads_started":1}
I20260924 10:46:05.786065 14214 heartbeater.cc:347] Connected to a master server at 127.2.182.62:33955
I20260924 10:46:05.786326 14214 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:05.786954 14214 heartbeater.cc:511] Master 127.2.182.62:33955 requested a full tablet report, sending...
I20260924 10:46:05.789201 14002 ts_manager.cc:194] Registered new tserver with Master: beefdb84b555430c89e36d8f76f83377 (127.2.182.1:36781)
I20260924 10:46:05.790711 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:39233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:46:05.802032 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:37985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:46:05.921980 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:05.922309 14222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:05.922405 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:05.927176 14222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:05.927260 14222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:05.927358 14222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:46:05.932632 14222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.182.20:37985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:05.933693 14222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:05.934706 14222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:05.944456 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:05.944638 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:05.945144 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:05.945559 14222 server_base.cc:1034] running on GCE node
I20260924 10:46:05.945988 14222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:05.946815 14222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:05.947993 14222 hybrid_clock.cc:648] HybridClock initialized: now 1790246765947965 us; error 49 us; skew 500 ppm
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:05.951030 14222 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:46:05.952148 14222 webserver.cc:460] Webserver started at http://127.2.182.2:42159/ using document root <none> and password file <none>
I20260924 10:46:05.952771 14222 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:05.952862 14222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:05.953109 14222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:05 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:05.970652 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "34b585f1de1c4d4082561baca0a7d895"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "06be8cc2d7c574261b341749b7b9ebad"
server_key_iv: "4368f41b4dbaaae4182388b457ee1b9a"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:05.971695 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "34b585f1de1c4d4082561baca0a7d895"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "06be8cc2d7c574261b341749b7b9ebad"
server_key_iv: "4368f41b4dbaaae4182388b457ee1b9a"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:05.976999 14222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 10:46:06.002522 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.004320 14222 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260924 10:46:06.004446 14222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "34b585f1de1c4d4082561baca0a7d895"
format_stamp: "Formatted at 2026-09-24 10:46:05 on dist-test-slave-6lwt"
server_key: "06be8cc2d7c574261b341749b7b9ebad"
server_key_iv: "4368f41b4dbaaae4182388b457ee1b9a"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:06.004576 14222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:06.025224 14222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:06.028977 14222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:06.029219 14222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:06.029909 14222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:06.030959 14222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:06.031076 14222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.031176 14222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:06.031239 14222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.045197 14222 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:41637
I20260924 10:46:06.046379 14222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:46:06.053428 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 14222
I20260924 10:46:06.053563 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:46:06.053869 2776 external_mini_cluster.cc:1428] Setting key 2c94a6e8fdef5e0c311e3d639d93c187
I20260924 10:46:06.047765 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:41637 every 8 connection(s)
Sep 24 10:46:06 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:06.065532 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.051500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48509 (local address 127.2.182.62:33955)
0924 10:46:06.051630 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.051635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.052126 (+ 491us) server_negotiation.cc:407] Connection header received
0924 10:46:06.053398 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.053402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.053463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.053525 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:06.055404 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.056165 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.057961 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.058330 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.058407 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.061391 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:06.061396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:06.061399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:06.061425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:06.063200 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.063805 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.063809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.063811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.063888 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.064261 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.064264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.064266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.064465 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:06.064541 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.065244 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.065370 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:46:06.067309 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.050403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:33955 (local address 127.2.182.2:48509)
0924 10:46:06.051965 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.052018 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.053141 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.054087 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.054097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.054485 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:06.055249 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.055270 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.056327 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.056333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.057824 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.057834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.058594 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.058598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.058805 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.059475 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:06.059495 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:06.061209 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:06.063360 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:06.063366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:06.063380 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:06.063693 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:06.063986 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:06.063988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:06.063990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:06.064159 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:06.064606 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:06.064612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:06.064957 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.065952 (+ 995us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.066277 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":132,"threads_started":1}
I20260924 10:46:06.068821 14352 heartbeater.cc:347] Connected to a master server at 127.2.182.62:33955
I20260924 10:46:06.069089 14352 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:06.069686 14352 heartbeater.cc:511] Master 127.2.182.62:33955 requested a full tablet report, sending...
I20260924 10:46:06.070840 14002 ts_manager.cc:194] Registered new tserver with Master: 34b585f1de1c4d4082561baca0a7d895 (127.2.182.2:41637)
I20260924 10:46:06.071586 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:48509
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:46:06.100591 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:37985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:06.261047 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:06.261497 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:06.261636 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:06.267776 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:06.267906 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:06.268049 14357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:46:06.275890 14357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.182.20:37985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:06.277446 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:06.278777 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:06.298120 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:06.298475 14357 server_base.cc:1034] running on GCE node
W20260924 10:46:06.298796 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:06.299371 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:06.299893 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:06.300637 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:06.301802 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790246766301733 us; error 83 us; skew 500 ppm
Sep 24 10:46:06 dist-test-slave-6lwt krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:06.305236 14357 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:46:06.306186 14357 webserver.cc:460] Webserver started at http://127.2.182.3:42133/ using document root <none> and password file <none>
I20260924 10:46:06.306831 14357 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:06.306895 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:06.307094 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:06 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:06.323741 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "33ec158665d24e84b9828de9e55e6b57"
format_stamp: "Formatted at 2026-09-24 10:46:06 on dist-test-slave-6lwt"
server_key: "adf72212212f0f8e0da86649d6e52165"
server_key_iv: "82d4f68088b62450570700a6f6cd2dc9"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:06.334525 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "33ec158665d24e84b9828de9e55e6b57"
format_stamp: "Formatted at 2026-09-24 10:46:06 on dist-test-slave-6lwt"
server_key: "adf72212212f0f8e0da86649d6e52165"
server_key_iv: "82d4f68088b62450570700a6f6cd2dc9"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:06.341050 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 10:46:06.360307 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.361982 14357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:46:06.362092 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "33ec158665d24e84b9828de9e55e6b57"
format_stamp: "Formatted at 2026-09-24 10:46:06 on dist-test-slave-6lwt"
server_key: "adf72212212f0f8e0da86649d6e52165"
server_key_iv: "82d4f68088b62450570700a6f6cd2dc9"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:06.362221 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:06.391649 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:06.395066 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:06.395299 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:06.395943 14357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:06.396989 14357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:06.397087 14357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.397188 14357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:06.397248 14357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:06.410058 14357 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:34301
I20260924 10:46:06.411209 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:46:06.412007 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:34301 every 8 connection(s)
I20260924 10:46:06.420799 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 14357
I20260924 10:46:06.421023 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:46:06.421331 2776 external_mini_cluster.cc:1428] Setting key 87dd08380b0525a427824c63fccf0b4f
Sep 24 10:46:06 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
Sep 24 10:46:06 dist-test-slave-6lwt krb5kdc[13526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:06.438772 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.414634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:46317 (local address 127.2.182.62:33955)
0924 10:46:06.414761 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.414765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.423748 (+ 8983us) server_negotiation.cc:407] Connection header received
0924 10:46:06.425126 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.425129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.425188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.425244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:06.426980 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.427783 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.429488 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.429851 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.429938 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.433392 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:06.433396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:06.433398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:06.433424 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:06.436413 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.437023 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.437026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.437028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.437134 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.437539 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.437542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.437544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.437751 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:06.437827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.438490 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.438611 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:46:06.440588 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.415077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:33955 (local address 127.2.182.3:46317)
0924 10:46:06.423582 (+ 8505us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.423643 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.424851 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.425731 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.425743 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.426103 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:06.426810 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.426832 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.427940 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.427943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.429347 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.429358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.430151 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.430155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.430347 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.431067 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:06.431089 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:06.433209 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:06.436538 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:06.436544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:06.436561 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:06.436885 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:06.437233 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:06.437236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:06.437238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:06.437416 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:06.437891 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:06.437898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:06.438225 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.439155 (+ 930us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.439487 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8249,"thread_start_us":136,"threads_started":1}
I20260924 10:46:06.442059 14486 heartbeater.cc:347] Connected to a master server at 127.2.182.62:33955
I20260924 10:46:06.442299 14486 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:06.442845 14486 heartbeater.cc:511] Master 127.2.182.62:33955 requested a full tablet report, sending...
I20260924 10:46:06.444188 14002 ts_manager.cc:194] Registered new tserver with Master: 33ec158665d24e84b9828de9e55e6b57 (127.2.182.3:34301)
I20260924 10:46:06.444913 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:46317
I20260924 10:46:06.448868 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:46:06.460775 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.451428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.2.182.62:33955)
0924 10:46:06.451555 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.451559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.451676 (+ 117us) server_negotiation.cc:407] Connection header received
0924 10:46:06.451844 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.451847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.451889 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.451935 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:06.452824 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.453617 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.454772 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.455120 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.455186 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.457042 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:06.457046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:06.457049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:06.457073 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:06.458968 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.459465 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.459469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.459471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.459550 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:06.459838 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:06.459841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:06.459843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:06.460040 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:06.460149 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.460495 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.460613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:46:06.465571 14002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:46:06.470163 14002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:46:06.492959 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.483491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:41637 (local address 127.0.0.1:35228)
0924 10:46:06.485329 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.485343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.485422 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.486214 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.486217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.486234 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:06.486355 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.486361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.487644 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.487647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.489279 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.489285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.492401 (+ 3116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.492404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.492520 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.492554 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.492654 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.492763 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":486,"thread_start_us":86,"threads_started":1}
I20260924 10:46:06.493641 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.483785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36781 (local address 127.0.0.1:44476)
0924 10:46:06.488590 (+ 4805us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.488605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.488693 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.489368 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.489371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.489388 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:06.489475 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.489481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.490899 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.490902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.491942 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.491953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.493244 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.493248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.493350 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.493376 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.493444 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.493506 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4747,"thread_start_us":78,"threads_started":1}
I20260924 10:46:06.494060 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.484303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.2.182.1:36781)
0924 10:46:06.484914 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.484920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.488793 (+ 3873us) server_negotiation.cc:407] Connection header received
0924 10:46:06.488875 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.488880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.489088 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.489228 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:06.489604 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.490761 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.492134 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.492936 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.493045 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.493180 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.493810 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.493907 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":89,"threads_started":1}
I20260924 10:46:06.495404 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.482647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.2.182.3:34301)
0924 10:46:06.482987 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.482999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.484573 (+ 1574us) server_negotiation.cc:407] Connection header received
0924 10:46:06.484655 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.484659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.484878 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.485088 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:06.485394 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.486626 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.489413 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.490967 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.491066 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.491217 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.495139 (+ 3922us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.495249 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260924 10:46:06.495819 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.483667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.2.182.2:41637)
0924 10:46:06.485641 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.485651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.485671 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:46:06.485736 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.485741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.485913 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.486060 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:06.486484 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.487517 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.491292 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.492111 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.492222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.492343 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.493531 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.493627 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1881,"thread_start_us":105,"threads_started":1}
I20260924 10:46:06.497300 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 (DEFAULT_TABLE table=test-table [id=87a5baf9b1c640e4bfe03313e7b6e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:06.498406 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804b7c4101d149df8a419fc95efe1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:06.498742 14285 tablet_service.cc:1467] Processing CreateTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 (DEFAULT_TABLE table=test-table [id=87a5baf9b1c640e4bfe03313e7b6e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:06.499716 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804b7c4101d149df8a419fc95efe1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:06.506261 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.482086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:34301 (local address 127.0.0.1:43160)
0924 10:46:06.483818 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.483866 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.484003 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.485073 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.485080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.485105 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:06.485263 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.485269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.487419 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.487422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.488828 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.488838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.494635 (+ 5797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.494638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.494738 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.494759 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.494823 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.494886 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"mutex_wait_us":33,"thread_start_us":101,"threads_started":1}
I20260924 10:46:06.509102 14420 tablet_service.cc:1467] Processing CreateTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 (DEFAULT_TABLE table=test-table [id=87a5baf9b1c640e4bfe03313e7b6e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:06.510092 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804b7c4101d149df8a419fc95efe1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:06.511229 14505 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Bootstrap starting.
I20260924 10:46:06.522529 14506 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Bootstrap starting.
I20260924 10:46:06.524148 14505 tablet_bootstrap.cc:654] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:06.524664 14507 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: Bootstrap starting.
I20260924 10:46:06.525275 14505 log.cc:824] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:06.526615 14506 tablet_bootstrap.cc:654] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:06.528249 14507 tablet_bootstrap.cc:654] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:06.529137 14506 log.cc:824] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:06.531222 14507 log.cc:824] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:06.533773 14506 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: No bootstrap required, opened a new log
I20260924 10:46:06.534017 14506 ts_tablet_manager.cc:1397] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 10:46:06.537865 14506 raft_consensus.cc:348] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.539532 14507 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: No bootstrap required, opened a new log
I20260924 10:46:06.539755 14507 ts_tablet_manager.cc:1397] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260924 10:46:06.545226 14507 raft_consensus.cc:348] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.545652 14506 raft_consensus.cc:374] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:06.545763 14506 raft_consensus.cc:729] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ec158665d24e84b9828de9e55e6b57, State: Initialized, Role: FOLLOWER
I20260924 10:46:06.546500 14506 consensus_queue.cc:260] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.546790 14507 raft_consensus.cc:374] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:06.547891 14506 ts_tablet_manager.cc:1428] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260924 10:46:06.548013 14507 raft_consensus.cc:729] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beefdb84b555430c89e36d8f76f83377, State: Initialized, Role: FOLLOWER
I20260924 10:46:06.549212 14507 consensus_queue.cc:260] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.550480 14505 tablet_bootstrap.cc:492] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: No bootstrap required, opened a new log
I20260924 10:46:06.550589 14486 heartbeater.cc:503] Master 127.2.182.62:33955 was elected leader, sending a full tablet report...
I20260924 10:46:06.550710 14505 ts_tablet_manager.cc:1397] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20260924 10:46:06.554641 14507 ts_tablet_manager.cc:1428] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 10:46:06.555042 14214 heartbeater.cc:503] Master 127.2.182.62:33955 was elected leader, sending a full tablet report...
I20260924 10:46:06.555213 14505 raft_consensus.cc:348] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.555482 14505 raft_consensus.cc:374] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:06.555550 14505 raft_consensus.cc:729] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b585f1de1c4d4082561baca0a7d895, State: Initialized, Role: FOLLOWER
I20260924 10:46:06.556051 14505 consensus_queue.cc:260] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.557178 14505 ts_tablet_manager.cc:1428] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:46:06.557557 14352 heartbeater.cc:503] Master 127.2.182.62:33955 was elected leader, sending a full tablet report...
I20260924 10:46:06.610875 14511 raft_consensus.cc:482] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:46:06.611132 14511 raft_consensus.cc:504] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.612577 14511 leader_election.cc:288] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34b585f1de1c4d4082561baca0a7d895 (127.2.182.2:41637), beefdb84b555430c89e36d8f76f83377 (127.2.182.1:36781)
I20260924 10:46:06.617376 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.612775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:41637 (local address 127.2.182.3:51521)
0924 10:46:06.612884 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.612921 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.613013 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.613785 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.613788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.613808 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:06.613934 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.613940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.614944 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.614947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.616083 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.616093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.617005 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.617009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.617100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.617120 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.617192 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.617254 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":8}
I20260924 10:46:06.617890 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.613204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:45871 (local address 127.2.182.1:36781)
0924 10:46:06.613318 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.613323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.613633 (+ 310us) server_negotiation.cc:407] Connection header received
0924 10:46:06.613821 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.613824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.613888 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.613937 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:06.614369 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.615081 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.616577 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.617180 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.617264 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.617320 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.617680 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.617755 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:46:06.617995 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.613422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:51521 (local address 127.2.182.2:41637)
0924 10:46:06.613529 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.613534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.613551 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:46:06.613611 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.613617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.613673 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.613754 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:06.614077 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.614824 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.616242 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.616777 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.616867 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.616952 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.617823 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.617891 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:46:06.618402 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.613073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36781 (local address 127.2.182.3:45871)
0924 10:46:06.613527 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.613542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:06.613697 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:06.614084 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:06.614087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:06.614103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:06.614225 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.614235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.615239 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.615243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.616421 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:06.616432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.617429 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.617433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:06.617529 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.617563 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:46:06.618179 (+ 616us) client_negotiation.cc:240] Negotiation successful
0924 10:46:06.618251 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":135,"threads_started":1}
I20260924 10:46:06.618631 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804b7c4101d149df8a419fc95efe1eb2" candidate_uuid: "33ec158665d24e84b9828de9e55e6b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b585f1de1c4d4082561baca0a7d895" is_pre_election: true
I20260924 10:46:06.618927 14304 raft_consensus.cc:2454] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ec158665d24e84b9828de9e55e6b57 in term 0.
I20260924 10:46:06.619167 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804b7c4101d149df8a419fc95efe1eb2" candidate_uuid: "33ec158665d24e84b9828de9e55e6b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefdb84b555430c89e36d8f76f83377" is_pre_election: true
I20260924 10:46:06.619396 14168 raft_consensus.cc:2454] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ec158665d24e84b9828de9e55e6b57 in term 0.
I20260924 10:46:06.619385 14375 leader_election.cc:302] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ec158665d24e84b9828de9e55e6b57, 34b585f1de1c4d4082561baca0a7d895; no voters:
I20260924 10:46:06.619756 14511 raft_consensus.cc:2781] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:46:06.619838 14511 raft_consensus.cc:482] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:46:06.619876 14511 raft_consensus.cc:3037] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:06.621255 14511 raft_consensus.cc:504] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.621606 14511 leader_election.cc:288] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [CANDIDATE]: Term 1 election: Requested vote from peers 34b585f1de1c4d4082561baca0a7d895 (127.2.182.2:41637), beefdb84b555430c89e36d8f76f83377 (127.2.182.1:36781)
I20260924 10:46:06.621942 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804b7c4101d149df8a419fc95efe1eb2" candidate_uuid: "33ec158665d24e84b9828de9e55e6b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beefdb84b555430c89e36d8f76f83377"
I20260924 10:46:06.622058 14168 raft_consensus.cc:3037] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:06.622062 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804b7c4101d149df8a419fc95efe1eb2" candidate_uuid: "33ec158665d24e84b9828de9e55e6b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b585f1de1c4d4082561baca0a7d895"
I20260924 10:46:06.622179 14304 raft_consensus.cc:3037] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:06.623361 14168 raft_consensus.cc:2454] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ec158665d24e84b9828de9e55e6b57 in term 1.
I20260924 10:46:06.623411 14304 raft_consensus.cc:2454] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ec158665d24e84b9828de9e55e6b57 in term 1.
I20260924 10:46:06.623801 14375 leader_election.cc:302] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ec158665d24e84b9828de9e55e6b57, 34b585f1de1c4d4082561baca0a7d895; no voters:
I20260924 10:46:06.624054 14511 raft_consensus.cc:2781] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:06.624285 14511 raft_consensus.cc:686] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 1 LEADER]: Becoming Leader. State: Replica: 33ec158665d24e84b9828de9e55e6b57, State: Running, Role: LEADER
I20260924 10:46:06.624631 14511 consensus_queue.cc:237] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } }
I20260924 10:46:06.628072 14002 catalog_manager.cc:5495] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33ec158665d24e84b9828de9e55e6b57 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "33ec158665d24e84b9828de9e55e6b57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec158665d24e84b9828de9e55e6b57" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 34301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 } health_report { overall_health: UNKNOWN } } }
I20260924 10:46:06.641659 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.637454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.2.182.3:34301)
0924 10:46:06.637608 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:06.637612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:06.637627 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:46:06.637696 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:06.637699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:06.637757 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:06.637872 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:46:06.638121 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.638903 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.640018 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:06.640334 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:06.640442 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:06.640724 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:46:06.641272 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:46:06.641395 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:06.641466 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:46:06.641521 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:46:06.648764 14168 raft_consensus.cc:1261] T 804b7c4101d149df8a419fc95efe1eb2 P beefdb84b555430c89e36d8f76f83377 [term 1 FOLLOWER]: Refusing update from remote peer 33ec158665d24e84b9828de9e55e6b57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:06.648859 14304 raft_consensus.cc:1261] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 1 FOLLOWER]: Refusing update from remote peer 33ec158665d24e84b9828de9e55e6b57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:06.649653 14511 consensus_queue.cc:1035] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34b585f1de1c4d4082561baca0a7d895" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:06.649881 14515 consensus_queue.cc:1035] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beefdb84b555430c89e36d8f76f83377" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:06.656427 14518 mvcc.cc:205] Tried to move back new op lower bound from 7332850756184309760 to 7332850756097331200. Current Snapshot: MvccSnapshot[applied={T|T < 7332850756184309760}]
I20260924 10:46:06.663290 14519 mvcc.cc:205] Tried to move back new op lower bound from 7332850756184309760 to 7332850756097331200. Current Snapshot: MvccSnapshot[applied={T|T < 7332850756184309760}]
I20260924 10:46:06.665132 14520 mvcc.cc:205] Tried to move back new op lower bound from 7332850756184309760 to 7332850756097331200. Current Snapshot: MvccSnapshot[applied={T|T < 7332850756184309760}]
I20260924 10:46:06.669607 14002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:46:06.669745 14002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:46:06.672109 14002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd59648ea4bc43c3849b9c5af5c876ba: Sending DeleteTablet for 3 replicas of tablet 804b7c4101d149df8a419fc95efe1eb2
I20260924 10:46:06.673029 14285 tablet_service.cc:1514] Processing DeleteTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:06 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:35228
I20260924 10:46:06.673126 14148 tablet_service.cc:1514] Processing DeleteTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:06 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44476
I20260924 10:46:06.674079 14420 tablet_service.cc:1514] Processing DeleteTablet for tablet 804b7c4101d149df8a419fc95efe1eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:06 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:43160
I20260924 10:46:06.675141 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 14083
I20260924 10:46:06.682317 14530 tablet_replica.cc:330] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: stopping tablet replica
I20260924 10:46:06.682636 14530 raft_consensus.cc:2229] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:46:06.683039 14530 raft_consensus.cc:2258] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:46:06.689098 14528 tablet_replica.cc:330] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: stopping tablet replica
I20260924 10:46:06.689440 14528 raft_consensus.cc:2229] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:46:06.689571 14530 ts_tablet_manager.cc:1905] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:46:06.689781 14528 raft_consensus.cc:2258] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:46:06.691082 14528 ts_tablet_manager.cc:1905] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:46:06.692159 14530 ts_tablet_manager.cc:1918] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:46:06.692245 14530 log.cc:1196] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/804b7c4101d149df8a419fc95efe1eb2
I20260924 10:46:06.692531 14530 ts_tablet_manager.cc:1939] T 804b7c4101d149df8a419fc95efe1eb2 P 33ec158665d24e84b9828de9e55e6b57: Deleting consensus metadata
I20260924 10:46:06.693511 14528 ts_tablet_manager.cc:1918] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:46:06.693599 14528 log.cc:1196] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/804b7c4101d149df8a419fc95efe1eb2
I20260924 10:46:06.693784 13988 catalog_manager.cc:4841] TS 33ec158665d24e84b9828de9e55e6b57 (127.2.182.3:34301): tablet 804b7c4101d149df8a419fc95efe1eb2 (table test-table [id=87a5baf9b1c640e4bfe03313e7b6e010]) successfully deleted
I20260924 10:46:06.693866 14528 ts_tablet_manager.cc:1939] T 804b7c4101d149df8a419fc95efe1eb2 P 34b585f1de1c4d4082561baca0a7d895: Deleting consensus metadata
W20260924 10:46:06.694057 13988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:36781 (error 108)
I20260924 10:46:06.694886 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 14222
I20260924 10:46:06.695464 13988 catalog_manager.cc:4841] TS 34b585f1de1c4d4082561baca0a7d895 (127.2.182.2:41637): tablet 804b7c4101d149df8a419fc95efe1eb2 (table test-table [id=87a5baf9b1c640e4bfe03313e7b6e010]) successfully deleted
I20260924 10:46:06.701229 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:06.700926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36781 (local address 127.0.0.1:44482)
0924 10:46:06.701090 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:06.701150 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 10:46:06.701470 13988 catalog_manager.cc:4584] TS beefdb84b555430c89e36d8f76f83377 (127.2.182.1:36781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 804b7c4101d149df8a419fc95efe1eb2: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36781: connect: Connection refused (error 111)
I20260924 10:46:06.703547 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 14357
I20260924 10:46:06.711633 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 13970
I20260924 10:46:06.717006 2776 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:46:07.051506 2776 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:46:07.051646 2776 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:46:07.395978 2776 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:46:07.396 UTC [13556] LOG: received smart shutdown request
2026-09-24 10:46:07.399 UTC [13556] LOG: background worker "logical replication launcher" (PID 13565) exited with exit code 1
2026-09-24 10:46:07.399 UTC [13560] LOG: shutting down
2026-09-24 10:46:07.558 UTC [13556] LOG: database system is shut down
2026-09-24T10:46:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43677 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:46:07 dist-test-slave-6lwt krb5kdc[14540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:46:07 dist-test-slave-6lwt krb5kdc[14540](info): set up 2 sockets
Sep 24 10:46:07 dist-test-slave-6lwt krb5kdc[14540](info): commencing operation
krb5kdc: starting...
W20260924 10:46:08.697072 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:46:08 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:46:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangeradmin_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangeradmin_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangerlookup_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangerlookup_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
I20260924 10:46:08.857208 2776 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwg6lLW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/postgres -l logfile start
2026-09-24 10:46:11.249 UTC [14570] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:46:11.249 UTC [14570] LOG: listening on IPv4 address "127.2.182.20", port 41281
2026-09-24 10:46:11.252 UTC [14570] LOG: listening on Unix socket "/tmp/.s.PGSQL.41281"
2026-09-24 10:46:11.286 UTC [14573] LOG: database system was shut down at 2026-09-24 10:46:09 UTC
2026-09-24 10:46:11.290 UTC [14570] LOG: database system is ready to accept connections
I20260924 10:46:12.220956 2776 mini_postgres.cc:96] Postgres bound to 41281
2026-09-24 10:46:12.227 UTC [14581] FATAL: database "slave" does not exist
127.2.182.20:41281 - accepting connections
I20260924 10:46:12.228008 2776 mini_ranger.cc:161] Starting Ranger...
I20260924 10:46:12.242544 2776 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:46:12.695334 2776 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:46:12.695405 2776 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin
2026-09-24 10:46:12,940 [I] DB FLAVOR :POSTGRES
2026-09-24 10:46:12,940 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:46:12,940 [I] Checking connection..
2026-09-24 10:46:12,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:46:13,202 [I] Checking connection passed.
2026-09-24 10:46:13,202 [I] --------- Verifying version history table ---------
2026-09-24 10:46:13,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:46:13,451 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:46:13,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:46:13,693 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:46:13,694 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:46:13,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:46:13.926 UTC [14645] WARNING: there is no transaction in progress
2026-09-24 10:46:13,932 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:46:13,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:46:14,154 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:46:14,154 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:46:14,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:46:14,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:46:14,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6lwt.c.gcp-upstream.internal','N') ;"
2026-09-24 10:46:14,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:46:15,128 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:46:15,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:46:15,383 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:46:15,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:46:15,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:46:15,852 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:46:15,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:46:16.834 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.851 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.868 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.882 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.898 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.915 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.968 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.974 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.989 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:16.998 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.008 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.021 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.031 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.037 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.044 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17.051 UTC [14753] WARNING: there is no transaction in progress
2026-09-24 10:46:17,689 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:46:17,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:46:17,919 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:46:17,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:46:18,170 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:46:18,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:46:18,417 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:46:18,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:46:18,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:46:18,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6lwt.c.gcp-upstream.internal';"
2026-09-24 10:46:19,136 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:46:19,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:46:19,381 [I] DB_PATCHES have already been applied
I20260924 10:46:19.387429 2776 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwg6lLW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:46:19.387532 2776 mini_ranger.cc:193] Using host: 127.2.182.20
I20260924 10:46:19.390887 2776 mini_ranger.cc:227] Ranger admin URL: http://127.2.182.20:45587
Sep 24, 2026 10:46:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:46:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:46:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:46:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:46:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:46:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:46:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:46:20.502254 2776 mini_ranger.cc:160] Time spent starting Ranger: real 8.274s user 0.000s sys 0.009s
Sep 24, 2026 10:46:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:46:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 10:46:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:46:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:46:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:46:41.008694 2776 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.182.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.182.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangerkms_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.182.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/rangerkms_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.182.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/HTTP_127.2.182.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:46:41 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246801, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:46:41.092792 2776 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:46:41.102821 2776 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:46:42.031457 2776 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:46:42.031530 2776 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms
I20260924 10:46:42.031543 2776 mini_ranger_kms.cc:227] Using postgres at 127.2.182.20:41281
2026-09-24 10:46:42,203 [I] DB FLAVOR :POSTGRES
2026-09-24 10:46:42,203 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:46:42,203 [I] Checking connection
2026-09-24 10:46:42,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:46:42,448 [I] connection success
2026-09-24 10:46:42,448 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:46:42,448 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:46:42,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:46:42,669 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:46:42,669 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:46:42,669 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:46:42,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.20:41281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:46:42,914 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:46:43.405889 2776 mini_ranger_kms.cc:310] Created kms service
I20260924 10:46:43.568773 2776 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:46:43.609740 2776 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:46:43 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246801, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:43.983090 2776 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:46:43.983239 2776 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwg6lLW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwg6lLW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:46:43.983263 2776 mini_ranger_kms.cc:240] Using host: 127.2.182.20
I20260924 10:46:43.987751 2776 mini_ranger_kms.cc:276] Ranger KMS PID: 14933
I20260924 10:46:43.987844 2776 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.182.20:42013
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:46:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:46:46.228685 2776 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.136s user 0.013s sys 0.142s
I20260924 10:46:46.228826 2776 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:46:46.228891 2776 mini_ranger_kms.cc:401] 127.2.182.20:42013/kms/v1/keys
Sep 24 10:46:46 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:46:46 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
Sep 24 10:46:47 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.20@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
Sep 24 10:46:49 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:46:49.264271 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43393
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:43393
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:49.371189 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:49.371454 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:49.371541 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:49.375517 14984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:49.375595 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:49.375626 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:49.375664 14984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:49.375701 14984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:49.380882 14984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.182.20:42013/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:43393
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:49.381887 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:49.382861 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:49.390929 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:49.390920 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:49.391254 14984 server_base.cc:1034] running on GCE node
W20260924 10:46:49.391309 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:49.391955 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:49.393195 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:49.394371 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790246809394350 us; error 36 us; skew 500 ppm
Sep 24 10:46:49 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246809, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:49.397169 14984 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:46:49.398125 14984 webserver.cc:460] Webserver started at http://127.2.182.62:44235/ using document root <none> and password file <none>
I20260924 10:46:49.398658 14984 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:49.398712 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:49.398934 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:49 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246809, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:49.459425 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "9507338246d848d792e4a53dddd34861"
format_stamp: "Formatted at 2026-09-24 10:46:49 on dist-test-slave-6lwt"
server_key: "379be4e0cfbf165a44ae62ceb320efc1"
server_key_iv: "5e9671d97bea5765ac36c8aeab141561"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:49.460105 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "9507338246d848d792e4a53dddd34861"
format_stamp: "Formatted at 2026-09-24 10:46:49 on dist-test-slave-6lwt"
server_key: "379be4e0cfbf165a44ae62ceb320efc1"
server_key_iv: "5e9671d97bea5765ac36c8aeab141561"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:49.464097 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:46:49.480546 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:49.482138 14984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:46:49.482285 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "9507338246d848d792e4a53dddd34861"
format_stamp: "Formatted at 2026-09-24 10:46:49 on dist-test-slave-6lwt"
server_key: "379be4e0cfbf165a44ae62ceb320efc1"
server_key_iv: "5e9671d97bea5765ac36c8aeab141561"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:49.482429 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:49.512981 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:49.516073 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:49.516583 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:49.524264 14984 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:43393
I20260924 10:46:49.524300 15051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:43393 every 8 connection(s)
I20260924 10:46:49.525353 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:46:49.528016 15052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:49.531272 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 14984
I20260924 10:46:49.531387 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:46:49.531682 2776 external_mini_cluster.cc:1428] Setting key 1db1cecae5953c706e8448e4990ac5eb
I20260924 10:46:49.534163 15052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Bootstrap starting.
I20260924 10:46:49.536712 15052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:49.537592 15052 log.cc:824] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:49.539547 15052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: No bootstrap required, opened a new log
Sep 24 10:46:49 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:49.543918 15052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:49.544157 15052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:49.544260 15052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9507338246d848d792e4a53dddd34861, State: Initialized, Role: FOLLOWER
I20260924 10:46:49.544746 15052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:49.544950 15052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:46:49.545050 15052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:46:49.545163 15052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:49.546175 15052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:49.546581 15052 leader_election.cc:302] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9507338246d848d792e4a53dddd34861; no voters:
I20260924 10:46:49.546967 15052 leader_election.cc:288] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:46:49.547348 15058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:49.547600 15058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 LEADER]: Becoming Leader. State: Replica: 9507338246d848d792e4a53dddd34861, State: Running, Role: LEADER
I20260924 10:46:49.547915 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:49.533128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.2.182.62:43393)
0924 10:46:49.533558 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:49.533576 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:49.533616 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:46:49.534432 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:49.534455 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:49.534864 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:49.535209 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:49.536052 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:49.537531 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:49.538734 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:49.539391 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:49.539545 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:49.542616 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:49.542622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:49.542638 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:49.542665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:49.544709 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:49.545238 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:49.545244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:49.545250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:49.545375 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:49.545699 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:49.545702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:49.545704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:49.546171 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:49.546370 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:49.546728 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 10:46:49.547004 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260924 10:46:49.547940 15058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:49.548346 15052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:49.549489 15058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9507338246d848d792e4a53dddd34861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } } }
I20260924 10:46:49.549690 15058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:49.549957 15058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9507338246d848d792e4a53dddd34861. Latest consensus state: current_term: 1 leader_uuid: "9507338246d848d792e4a53dddd34861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } } }
I20260924 10:46:49.550120 15058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:49.550414 15066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:46:49.550613 15052 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:46:49.557683 15066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:46:49.570494 15066 catalog_manager.cc:1347] Generated new cluster ID: 2f85b9c263d94469a102e09fa1ba94c5
I20260924 10:46:49.570560 15066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:46:49.580024 15066 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:46:49.581318 15066 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:46:49.590834 15066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Generated new TSK 0
I20260924 10:46:49.591691 15066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:46:50.320636 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:50.443109 15097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:50.443343 15097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:50.443401 15097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:50.447461 15097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:50.447507 15097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:50.447559 15097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:46:50.452873 15097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:50.454022 15097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:50.455312 15097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:50.466547 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:50.466562 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:50.466532 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:50.466975 15097 server_base.cc:1034] running on GCE node
I20260924 10:46:50.467370 15097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:50.467983 15097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:50.469165 15097 hybrid_clock.cc:648] HybridClock initialized: now 1790246810469136 us; error 40 us; skew 500 ppm
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:50.472012 15097 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:46:50.472800 15097 webserver.cc:460] Webserver started at http://127.2.182.1:44043/ using document root <none> and password file <none>
I20260924 10:46:50.473402 15097 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:50.473495 15097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:50.473748 15097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:50.486299 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "209826dee5704954af2aa96c5198fbaa"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "bc7b7c79c50fac872490157726112bbe"
server_key_iv: "8f4a3ae22938eed1bdb50fc0c5532647"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.487200 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "209826dee5704954af2aa96c5198fbaa"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "bc7b7c79c50fac872490157726112bbe"
server_key_iv: "8f4a3ae22938eed1bdb50fc0c5532647"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.491750 15097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:46:50.510841 15114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.512702 15097 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260924 10:46:50.512847 15097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "209826dee5704954af2aa96c5198fbaa"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "bc7b7c79c50fac872490157726112bbe"
server_key_iv: "8f4a3ae22938eed1bdb50fc0c5532647"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.513031 15097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:50.536244 15097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:50.539901 15097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:50.540129 15097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:50.540836 15097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:50.541883 15097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:50.541993 15097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.542128 15097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:50.542217 15097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.555313 15097 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40503
I20260924 10:46:50.556478 15097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:46:50.557472 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40503 every 8 connection(s)
I20260924 10:46:50.558060 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15097
I20260924 10:46:50.558171 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:46:50.558473 2776 external_mini_cluster.cc:1428] Setting key 96515653ef2586ad0eba3f5d0c3b0194
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:50.607148 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:50.564279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:59929 (local address 127.2.182.62:43393)
0924 10:46:50.577103 (+ 12824us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:50.577115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:50.577131 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:46:50.577175 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:50.577177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:50.577218 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:50.577340 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:50.578892 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.579803 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.581637 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.582010 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.582108 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:50.586254 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:50.586261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:50.586267 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:50.586300 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:50.588443 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:50.604736 (+ 16293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:50.604741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:50.604743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:50.604907 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:50.605463 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:50.605467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:50.605469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:50.605720 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:50.605825 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:50.606807 (+ 982us) server_negotiation.cc:299] Negotiation successful
0924 10:46:50.606962 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12738,"thread_start_us":104,"threads_started":1}
I20260924 10:46:50.609337 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:50.563212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.1:59929)
0924 10:46:50.563746 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:50.563798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:50.564867 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:50.577573 (+ 12706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:50.577584 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:50.577960 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:50.578675 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:50.578697 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.579968 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.579971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:50.581489 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:50.581501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.582329 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.582332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:50.582468 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:50.583116 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:50.583157 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:50.586026 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:50.589343 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:50.589350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:50.589369 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:50.604398 (+ 15029us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:50.605049 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:50.605052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:50.605054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:50.605293 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:50.606005 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:50.606012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:50.606433 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 10:46:50.607507 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0924 10:46:50.608076 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":133,"threads_started":1}
I20260924 10:46:50.610844 15228 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:50.611198 15228 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:50.611878 15228 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:46:50.613644 15016 ts_manager.cc:194] Registered new tserver with Master: 209826dee5704954af2aa96c5198fbaa (127.2.182.1:40503)
I20260924 10:46:50.613684 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:46:50.615195 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:59929
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:46:50.806957 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:50.807428 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:50.807538 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:50.813292 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:50.813433 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:50.813561 15236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:46:50.821144 15236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:50.822662 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:50.824025 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:50.836843 15242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:50.837077 15243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:50.837340 15236 server_base.cc:1034] running on GCE node
W20260924 10:46:50.837625 15245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:50.838132 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:50.838907 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:50.840054 15236 hybrid_clock.cc:648] HybridClock initialized: now 1790246810839989 us; error 82 us; skew 500 ppm
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:50.843636 15236 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:46:50.844657 15236 webserver.cc:460] Webserver started at http://127.2.182.2:40405/ using document root <none> and password file <none>
I20260924 10:46:50.845355 15236 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:50.845423 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:50.845633 15236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:50.862462 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "1aa4698e54614eb28f72742e0b459056"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "393cc01eb862509028f7103602f5787c"
server_key_iv: "f062542cf55d8ea8510e27ec8046f7ce"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.865674 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "1aa4698e54614eb28f72742e0b459056"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "393cc01eb862509028f7103602f5787c"
server_key_iv: "f062542cf55d8ea8510e27ec8046f7ce"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.870836 15236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:46:50.894516 15252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.896164 15236 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260924 10:46:50.896288 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "1aa4698e54614eb28f72742e0b459056"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "393cc01eb862509028f7103602f5787c"
server_key_iv: "f062542cf55d8ea8510e27ec8046f7ce"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:50.896409 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:50.921021 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:50.924602 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:50.924852 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:50.925560 15236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:50.926605 15236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:50.926708 15236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.926815 15236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:50.926877 15236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:50.939941 15236 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42011
I20260924 10:46:50.941289 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:46:50.942261 15365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:42011 every 8 connection(s)
I20260924 10:46:50.949869 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15236
I20260924 10:46:50.949994 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:46:50.950302 2776 external_mini_cluster.cc:1428] Setting key 1316ea3492487aba02dd3a1c28df5256
Sep 24 10:46:50 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:50.959291 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:50.945983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38391 (local address 127.2.182.62:43393)
0924 10:46:50.946106 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:50.946111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:50.946620 (+ 509us) server_negotiation.cc:407] Connection header received
0924 10:46:50.947817 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:50.947821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:50.947875 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:50.947932 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:50.949619 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.950349 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.951968 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.952263 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.952333 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:50.955234 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:50.955237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:50.955241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:50.955281 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:50.957011 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:50.957572 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:50.957575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:50.957577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:50.957654 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:50.958044 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:50.958047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:50.958049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:50.958254 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:50.958330 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:50.958997 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 10:46:50.959122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:46:50.961227 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:50.944913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.2:38391)
0924 10:46:50.946466 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:50.946521 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:50.947559 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:50.948383 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:50.948393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:50.948769 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:50.949457 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:50.949475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.950576 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.950579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:50.951837 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:50.951847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:50.952513 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:50.952516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:50.952697 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:50.953405 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:50.953425 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:50.955051 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:50.957133 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:50.957139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:50.957155 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:50.957463 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:50.957762 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:50.957764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:50.957766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:50.957942 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:50.958393 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:50.958399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:50.958711 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 10:46:50.959843 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0924 10:46:50.960193 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":148,"threads_started":1}
I20260924 10:46:50.962684 15366 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:50.962941 15366 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:50.963485 15366 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:46:50.964564 15016 ts_manager.cc:194] Registered new tserver with Master: 1aa4698e54614eb28f72742e0b459056 (127.2.182.2:42011)
I20260924 10:46:50.965302 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:38391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:46:50.987936 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:51.168761 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:51.169169 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:51.169276 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:51.175356 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:51.175434 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:51.175521 15371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:46:51.183429 15371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:51.184934 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:51.186401 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:51.199951 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:51.200560 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:51.201661 15371 server_base.cc:1034] running on GCE node
W20260924 10:46:51.202029 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:51.202539 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:51.203240 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246810, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:51.206650 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790246811206186 us; error 469 us; skew 500 ppm
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:51.213861 15371 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:46:51.215030 15371 webserver.cc:460] Webserver started at http://127.2.182.3:35009/ using document root <none> and password file <none>
I20260924 10:46:51.215700 15371 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:51.215802 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:51.216032 15371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:51.232169 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "a5503b5ee2214fb0b98269437099c965"
format_stamp: "Formatted at 2026-09-24 10:46:51 on dist-test-slave-6lwt"
server_key: "0112c5c9003b2011eb908061f74341f1"
server_key_iv: "f52a15e12a951a15bb4a1a997ce73365"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:51.232934 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "a5503b5ee2214fb0b98269437099c965"
format_stamp: "Formatted at 2026-09-24 10:46:51 on dist-test-slave-6lwt"
server_key: "0112c5c9003b2011eb908061f74341f1"
server_key_iv: "f52a15e12a951a15bb4a1a997ce73365"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:51.238126 15371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 10:46:51.275090 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.276911 15371 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.001s sys 0.000s
I20260924 10:46:51.277047 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "a5503b5ee2214fb0b98269437099c965"
format_stamp: "Formatted at 2026-09-24 10:46:51 on dist-test-slave-6lwt"
server_key: "0112c5c9003b2011eb908061f74341f1"
server_key_iv: "f52a15e12a951a15bb4a1a997ce73365"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:51.277189 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:51.323896 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:51.327571 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:51.327833 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:51.328544 15371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:51.329635 15371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:51.329733 15371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.329814 15371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:51.329861 15371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.340852 15371 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44365
I20260924 10:46:51.342001 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:46:51.343129 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44365 every 8 connection(s)
I20260924 10:46:51.344013 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15371
I20260924 10:46:51.344126 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:46:51.344470 2776 external_mini_cluster.cc:1428] Setting key 2b38efe32a110a3bc1baaa4bdd696bdb
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:51.360448 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:51.345526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:53065 (local address 127.2.182.62:43393)
0924 10:46:51.345643 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:51.345647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:51.346055 (+ 408us) server_negotiation.cc:407] Connection header received
0924 10:46:51.347276 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:51.347280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:51.347332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:51.347420 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:51.348881 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.349708 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.351257 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.351556 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.351620 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:51.354476 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:51.354480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:51.354483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:51.354507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:51.356321 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:51.358577 (+ 2256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:51.358581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:51.358583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:51.358674 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:51.359196 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:51.359200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:51.359201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:51.359407 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:51.359517 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:51.360162 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 10:46:51.360288 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:46:51.361474 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:51.344460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.3:53065)
0924 10:46:51.345914 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:51.345967 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:51.347018 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:51.347621 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:51.347632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:51.348034 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:51.348682 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:51.348704 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.349853 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.349856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:51.351125 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:51.351144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.351799 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.351802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:51.351981 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:51.352604 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:51.352623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:51.354292 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:51.356456 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:51.356462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:51.356476 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:51.358454 (+ 1978us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:51.358806 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:51.358809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:51.358811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:51.358976 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:51.359550 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:51.359556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:51.359879 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 10:46:51.360089 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 10:46:51.360419 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":134,"threads_started":1}
I20260924 10:46:51.362552 15500 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:51.362782 15500 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:51.363314 15500 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:51.364378 15016 ts_manager.cc:194] Registered new tserver with Master: a5503b5ee2214fb0b98269437099c965 (127.2.182.3:44365)
I20260924 10:46:51.365142 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:53065
I20260924 10:46:51.370849 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:46:51.371706 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15097
I20260924 10:46:51.379827 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15236
I20260924 10:46:51.385774 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15371
I20260924 10:46:51.392309 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 14984
I20260924 10:46:51.398877 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43393
--webserver_interface=127.2.182.62
--webserver_port=44235
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:43393
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:51.507004 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:51.507241 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:51.507310 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:51.511296 15503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:51.511348 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:51.511366 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:51.511399 15503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:51.511415 15503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:51.516659 15503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:43393
--ranger_config_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=44235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:51.517637 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:51.518615 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:51.526767 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:51.526816 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:51.526772 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:51.526960 15503 server_base.cc:1034] running on GCE node
I20260924 10:46:51.527596 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:51.528602 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:51.529783 15503 hybrid_clock.cc:648] HybridClock initialized: now 1790246811529770 us; error 28 us; skew 500 ppm
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:51.532469 15503 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:46:51.533387 15503 webserver.cc:460] Webserver started at http://127.2.182.62:44235/ using document root <none> and password file <none>
I20260924 10:46:51.533860 15503 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:51.533921 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:51.602283 15503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:46:51.604645 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.606068 15503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:46:51.606238 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "9507338246d848d792e4a53dddd34861"
format_stamp: "Formatted at 2026-09-24 10:46:49 on dist-test-slave-6lwt"
server_key: "379be4e0cfbf165a44ae62ceb320efc1"
server_key_iv: "5e9671d97bea5765ac36c8aeab141561"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:51.606853 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:51.646979 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:51.649664 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:51.649914 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:51.657869 15503 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:43393
I20260924 10:46:51.657899 15571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:43393 every 8 connection(s)
I20260924 10:46:51.658931 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:46:51.663065 15572 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:46:51.664645 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Bootstrap starting.
I20260924 10:46:51.666949 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15503
I20260924 10:46:51.667374 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:40503
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:43393
--webserver_port=44043
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:46:51.680647 15572 log.cc:824] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:51.687031 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:46:51.687425 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Bootstrap complete.
I20260924 10:46:51.691560 15572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:51.691921 15572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9507338246d848d792e4a53dddd34861, State: Initialized, Role: FOLLOWER
I20260924 10:46:51.692416 15572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:51.692528 15572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:46:51.692610 15572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:46:51.692719 15572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:46:51.694577 15572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:51.694900 15572 leader_election.cc:302] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9507338246d848d792e4a53dddd34861; no voters:
I20260924 10:46:51.695205 15572 leader_election.cc:288] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:46:51.695288 15576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:46:51.695487 15576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [term 2 LEADER]: Becoming Leader. State: Replica: 9507338246d848d792e4a53dddd34861, State: Running, Role: LEADER
I20260924 10:46:51.695780 15576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } }
I20260924 10:46:51.696403 15572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:51.696673 15576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9507338246d848d792e4a53dddd34861. Latest consensus state: current_term: 2 leader_uuid: "9507338246d848d792e4a53dddd34861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } } }
I20260924 10:46:51.696795 15576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:51.697230 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9507338246d848d792e4a53dddd34861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9507338246d848d792e4a53dddd34861" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 43393 } } }
I20260924 10:46:51.697423 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:51.698446 15580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:46:51.698843 15572 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:46:51.705096 15580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:46:51.705911 15580 catalog_manager.cc:1259] Loaded cluster ID: 2f85b9c263d94469a102e09fa1ba94c5
I20260924 10:46:51.705976 15580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:46:51.708520 15580 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:46:51.709298 15580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Loaded TSK: 0
I20260924 10:46:51.710623 15580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:46:51.806947 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:51.807209 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:51.807279 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:51.812300 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:51.812373 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:51.812469 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:46:51.819332 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:40503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=44043
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:51.820632 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:51.821720 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:51.831563 15610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:51.831682 15612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:51.831563 15609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:51.832576 15574 server_base.cc:1034] running on GCE node
I20260924 10:46:51.833098 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:51.833850 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:51.835027 15574 hybrid_clock.cc:648] HybridClock initialized: now 1790246811834998 us; error 42 us; skew 500 ppm
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:51.837860 15574 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:46:51.838919 15574 webserver.cc:460] Webserver started at http://127.2.182.1:44043/ using document root <none> and password file <none>
I20260924 10:46:51.839628 15574 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:51.839735 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:51.861093 15574 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:46:51.875715 15620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.877446 15574 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20260924 10:46:51.877583 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "209826dee5704954af2aa96c5198fbaa"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "bc7b7c79c50fac872490157726112bbe"
server_key_iv: "8f4a3ae22938eed1bdb50fc0c5532647"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:51.878278 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:51.900615 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:51.904342 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:51.904578 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:51.905309 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:51.906339 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:51.906402 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.906456 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:51.906487 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:51.918159 15574 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:40503
I20260924 10:46:51.919396 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:46:51.920200 15733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:40503 every 8 connection(s)
I20260924 10:46:51.925051 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15574
I20260924 10:46:51.925551 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:42011
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:43393
--webserver_port=40405
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:46:51 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246811, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:51.941756 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:51.922900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.1:46427)
0924 10:46:51.923508 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:51.923580 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:51.924634 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:51.926877 (+ 2243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:51.926887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:51.927230 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:51.927926 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:51.927946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.929879 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.929883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:51.931327 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:51.931339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.932657 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.932661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:51.932857 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:51.933609 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:51.933631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:51.935489 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:51.937961 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:51.937967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:51.937982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:51.938338 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:51.938758 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:51.938761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:51.938763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:51.938945 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:51.939779 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:51.939785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:51.940097 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 10:46:51.940317 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 10:46:51.940638 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260924 10:46:51.943588 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:51.922798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:46427 (local address 127.2.182.62:43393)
0924 10:46:51.925145 (+ 2347us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:51.925168 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:51.925213 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:46:51.925505 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:51.925521 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:51.925964 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:51.926320 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:51.928130 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.929713 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.931486 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:51.932250 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:51.932421 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:51.935690 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:51.935696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:51.935719 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:51.935748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:51.937819 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:51.938483 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:51.938490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:51.938495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:51.938639 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:51.939068 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:51.939072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:51.939073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:51.939580 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:51.939706 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:51.942314 (+ 2608us) server_negotiation.cc:299] Negotiation successful
0924 10:46:51.942607 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2184,"thread_start_us":126,"threads_started":1}
I20260924 10:46:51.947319 15734 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:51.947570 15734 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:51.948159 15734 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:51.950419 15734 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:43393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:46:51.950542 15734 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:51.950740 15734 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:51.951753 15734 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:51.951924 15734 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:51.952733 15734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:46:52.087725 15738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:52.088070 15738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:52.088152 15738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:52.092530 15738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:52.092633 15738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:52.092743 15738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:46:52.098928 15738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:42011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=40405
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:52.100096 15738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:52.101231 15738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:52.111807 15744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:52.111770 15743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:52.111881 15746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:52.112481 15738 server_base.cc:1034] running on GCE node
I20260924 10:46:52.112860 15738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:52.113528 15738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:52.114727 15738 hybrid_clock.cc:648] HybridClock initialized: now 1790246812114687 us; error 44 us; skew 500 ppm
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:52.118319 15738 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:46:52.119391 15738 webserver.cc:460] Webserver started at http://127.2.182.2:40405/ using document root <none> and password file <none>
I20260924 10:46:52.119951 15738 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:52.120023 15738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:52.140203 15738 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:46:52.143388 15754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.144997 15738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:46:52.145185 15738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "1aa4698e54614eb28f72742e0b459056"
format_stamp: "Formatted at 2026-09-24 10:46:50 on dist-test-slave-6lwt"
server_key: "393cc01eb862509028f7103602f5787c"
server_key_iv: "f062542cf55d8ea8510e27ec8046f7ce"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:52.145999 15738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:52.167958 15738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:52.171577 15738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:52.171823 15738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:52.172541 15738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:52.173621 15738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:52.173730 15738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.173837 15738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:52.173906 15738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.185011 15738 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42011
I20260924 10:46:52.185653 15867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:42011 every 8 connection(s)
I20260924 10:46:52.186297 15738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:46:52.195607 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15738
I20260924 10:46:52.196108 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.20:42013/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:44365
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:43393
--webserver_port=35009
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:52.204383 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:52.188887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.2:37049)
0924 10:46:52.189466 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:52.189525 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:52.190616 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:52.191238 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:52.191248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:52.191625 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:52.192230 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:52.192246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.193446 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.193450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:52.194881 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:52.194893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.195762 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.195766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:52.195971 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:52.196920 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:52.196941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:52.198961 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:52.201174 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:52.201180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:52.201194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:52.201534 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:52.201858 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:52.201862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:52.201863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:52.202034 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:52.202534 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:52.202540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:52.202847 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 10:46:52.203049 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 10:46:52.203394 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":109,"threads_started":1}
I20260924 10:46:52.205017 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:52.188503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:37049 (local address 127.2.182.62:43393)
0924 10:46:52.188683 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:52.188688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:52.189677 (+ 989us) server_negotiation.cc:407] Connection header received
0924 10:46:52.190864 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:52.190868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:52.190928 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:52.191038 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:52.192400 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.193286 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.195027 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.195419 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.195532 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:52.199153 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:52.199158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:52.199161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:52.199195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:52.201042 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.201656 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.201659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.201661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.201747 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.202154 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.202157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.202159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.202388 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:52.202481 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:52.204701 (+ 2220us) server_negotiation.cc:299] Negotiation successful
0924 10:46:52.204839 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:46:52.206493 15868 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:52.206763 15868 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.207346 15868 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:52.209278 15868 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:43393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:46:52.209398 15868 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.209545 15868 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:52.210413 15868 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.210564 15868 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:52.211333 15868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:46:52.339701 15871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:52.339994 15871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:52.340075 15871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:52.344501 15871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:52.344581 15871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:52.344681 15871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:46:52.352017 15871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.20:42013/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:44365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=35009
--tserver_master_addrs=127.2.182.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:52.353652 15871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:52.355029 15871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:52.366047 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:52.366400 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:52.366724 15871 server_base.cc:1034] running on GCE node
W20260924 10:46:52.366884 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:52.367305 15871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:52.367957 15871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:52.369167 15871 hybrid_clock.cc:648] HybridClock initialized: now 1790246812369136 us; error 47 us; skew 500 ppm
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:52.372022 15871 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:46:52.373008 15871 webserver.cc:460] Webserver started at http://127.2.182.3:35009/ using document root <none> and password file <none>
I20260924 10:46:52.373570 15871 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:52.373667 15871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:52.395875 15871 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260924 10:46:52.399142 15888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.400507 15871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:46:52.400637 15871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "a5503b5ee2214fb0b98269437099c965"
format_stamp: "Formatted at 2026-09-24 10:46:51 on dist-test-slave-6lwt"
server_key: "0112c5c9003b2011eb908061f74341f1"
server_key_iv: "f52a15e12a951a15bb4a1a997ce73365"
server_key_version: "kuduclusterkey@0"
I20260924 10:46:52.401288 15871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:52.415055 15871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:52.418010 15871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:52.418195 15871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:52.418875 15871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:46:52.419809 15871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:46:52.419915 15871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.420015 15871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:46:52.420082 15871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:52.431923 15871 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44365
I20260924 10:46:52.433061 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44365 every 8 connection(s)
I20260924 10:46:52.433108 15871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:46:52.437355 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 15871
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:52.452544 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:52.437161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:36945 (local address 127.2.182.62:43393)
0924 10:46:52.437328 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:52.437334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:52.437353 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:46:52.437997 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:52.438000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:52.438056 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:52.438114 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:52.439380 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.439976 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.441543 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.441772 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.441894 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:52.445824 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:52.445827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:52.445830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:52.445874 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:52.447475 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.449285 (+ 1810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.449289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.449290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.449373 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.449803 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.449806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.449808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.450012 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:52.450108 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:52.452265 (+ 2157us) server_negotiation.cc:299] Negotiation successful
0924 10:46:52.452387 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:46:52.453334 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:52.438031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.2.182.62:43393)
0924 10:46:52.438656 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:52.438661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:52.438674 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:46:52.438711 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:52.438713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:52.438752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:52.438835 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:52.439835 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.440595 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.441847 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.442149 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.442254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:52.445436 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:52.445440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:52.445443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:52.445467 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:52.448740 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.450690 (+ 1950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.450693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.450695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.450775 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:52.451092 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:52.451095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:52.451097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:52.451292 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:52.451485 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:52.453066 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0924 10:46:52.453185 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":105,"threads_started":1}
I20260924 10:46:52.454535 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:52.435826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:43393 (local address 127.2.182.3:36945)
0924 10:46:52.436425 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:52.436479 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:52.437724 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:52.438446 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:52.438452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:52.438794 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:52.439243 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:52.439255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.440092 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.440099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:52.441418 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:52.441428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:52.442036 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:52.442040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:52.442232 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:52.442946 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:52.442972 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:52.445275 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:52.447605 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:52.447611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:52.447627 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:52.447959 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:52.449484 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:52.449487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:52.449489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:52.449685 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:52.450176 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:52.450181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:52.450482 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 10:46:52.452208 (+ 1726us) client_negotiation.cc:240] Negotiation successful
0924 10:46:52.452494 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":164,"threads_started":1}
I20260924 10:46:52.455713 16002 heartbeater.cc:347] Connected to a master server at 127.2.182.62:43393
I20260924 10:46:52.455946 16002 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.456462 16002 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:52.458065 16002 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:43393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:46:52.458179 16002 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.458362 16002 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:52.459201 16002 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.459355 16002 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
W20260924 10:46:52.460124 16002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:46:52 dist-test-slave-6lwt krb5kdc[14540](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:52.953004 15734 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:52.953356 15734 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:52.954661 15535 ts_manager.cc:194] Registered new tserver with Master: 209826dee5704954af2aa96c5198fbaa (127.2.182.1:40503)
I20260924 10:46:52.956372 15535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:46427
Sep 24 10:46:53 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:53.211542 15868 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:53.211838 15868 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:53.212541 15535 ts_manager.cc:194] Registered new tserver with Master: 1aa4698e54614eb28f72742e0b459056 (127.2.182.2:42011)
I20260924 10:46:53.213414 15535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:37049
Sep 24 10:46:53 dist-test-slave-6lwt krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246812, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for HTTP/127.2.182.20@KRBTEST.COM
I20260924 10:46:53.460331 16002 heartbeater.cc:464] Registering TS with master...
I20260924 10:46:53.460659 16002 heartbeater.cc:511] Master 127.2.182.62:43393 requested a full tablet report, sending...
I20260924 10:46:53.461417 15535 ts_manager.cc:194] Registered new tserver with Master: a5503b5ee2214fb0b98269437099c965 (127.2.182.3:44365)
I20260924 10:46:53.462209 15535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:36945
I20260924 10:46:53.464814 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:46:53.475876 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.466391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.2.182.62:43393)
0924 10:46:53.466656 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.466661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.466989 (+ 328us) server_negotiation.cc:407] Connection header received
0924 10:46:53.467028 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.467031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.467075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.467146 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:53.468012 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.468795 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.469947 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.470282 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.470369 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.472006 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:53.472009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:53.472012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:53.472035 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:53.473753 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:53.474216 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:53.474238 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:53.474239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:53.474325 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:53.474635 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:53.474650 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:53.474652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:53.474852 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:53.474976 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.475578 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.475709 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260924 10:46:53.480113 15535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:46:53.485460 15535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:46:53.515019 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.505252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44365 (local address 127.0.0.1:51370)
0924 10:46:53.507322 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.507335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:53.507417 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:53.509276 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:53.509282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:53.509306 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:53.509452 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.509459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.511068 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.511071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.513264 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.513274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.514477 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.514481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.514585 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.514625 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:46:53.514735 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:46:53.514855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2007,"thread_start_us":86,"threads_started":1}
I20260924 10:46:53.518829 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.505447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.2.182.3:44365)
0924 10:46:53.507901 (+ 2454us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.507909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.507933 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:46:53.508010 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.508015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.508262 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.508415 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:53.509688 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.510922 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.513396 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.514155 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.514273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.514415 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.518597 (+ 4182us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.518694 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":93,"threads_started":1}
I20260924 10:46:53.521737 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.505876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42011 (local address 127.0.0.1:34450)
0924 10:46:53.507589 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.507602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:53.507675 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:53.517631 (+ 9956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:53.517634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:53.517651 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:53.517744 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.517750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.519243 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.519247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.520334 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.520346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.521388 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.521392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.521484 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.521508 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:46:53.521555 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 10:46:53.521613 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":91,"threads_started":1}
I20260924 10:46:53.522133 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.503746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40503 (local address 127.0.0.1:60556)
0924 10:46:53.506867 (+ 3121us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.506919 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:53.507062 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:53.516999 (+ 9937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:53.517002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:53.517020 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:53.517115 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.517121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.519210 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.519213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.520827 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.520837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.521783 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.521786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.521861 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.521881 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:46:53.521923 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 10:46:53.521982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2939,"thread_start_us":104,"threads_started":1}
I20260924 10:46:53.522003 15936 tablet_service.cc:1467] Processing CreateTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c (DEFAULT_TABLE table=test-table [id=479b0c3fe1404358a6ad1b3f11900cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:53.522759 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.506054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.2.182.2:42011)
0924 10:46:53.517157 (+ 11103us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.517164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.517189 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:46:53.517268 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.517273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.517463 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.517890 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:53.518037 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.519086 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.520484 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.521262 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.522356 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.522496 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.522543 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.522623 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10982,"thread_start_us":71,"threads_started":1}
I20260924 10:46:53.522785 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.504067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.2.182.1:40503)
0924 10:46:53.516483 (+ 12416us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.516491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.516516 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:46:53.516594 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.516599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.516809 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.517284 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:53.517437 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.518458 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.520981 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.521656 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.522386 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.522524 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.522574 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.522659 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12300,"thread_start_us":97,"threads_started":1}
I20260924 10:46:53.525364 15802 tablet_service.cc:1467] Processing CreateTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c (DEFAULT_TABLE table=test-table [id=479b0c3fe1404358a6ad1b3f11900cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:53.525519 15668 tablet_service.cc:1467] Processing CreateTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c (DEFAULT_TABLE table=test-table [id=479b0c3fe1404358a6ad1b3f11900cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:46:53.526384 15802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135ecbfaa9a43e59431f880c01c4e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:53.526521 15668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135ecbfaa9a43e59431f880c01c4e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:53.536178 15936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135ecbfaa9a43e59431f880c01c4e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:53.545267 16037 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Bootstrap starting.
I20260924 10:46:53.547583 16037 tablet_bootstrap.cc:654] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:53.548453 16037 log.cc:824] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:53.558697 16038 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Bootstrap starting.
I20260924 10:46:53.561275 16038 tablet_bootstrap.cc:654] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:53.562275 16038 log.cc:824] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:53.562812 16037 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: No bootstrap required, opened a new log
I20260924 10:46:53.563005 16037 ts_tablet_manager.cc:1397] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260924 10:46:53.564945 16038 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: No bootstrap required, opened a new log
I20260924 10:46:53.565127 16038 ts_tablet_manager.cc:1397] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:46:53.567099 16037 raft_consensus.cc:348] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.567348 16037 raft_consensus.cc:374] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:53.567399 16037 raft_consensus.cc:729] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa4698e54614eb28f72742e0b459056, State: Initialized, Role: FOLLOWER
I20260924 10:46:53.568111 16037 consensus_queue.cc:260] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.569267 16037 ts_tablet_manager.cc:1428] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:46:53.569679 15868 heartbeater.cc:503] Master 127.2.182.62:43393 was elected leader, sending a full tablet report...
I20260924 10:46:53.569995 16038 raft_consensus.cc:348] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.570256 16038 raft_consensus.cc:374] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:53.570320 16038 raft_consensus.cc:729] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209826dee5704954af2aa96c5198fbaa, State: Initialized, Role: FOLLOWER
I20260924 10:46:53.570751 16038 consensus_queue.cc:260] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.571695 16038 ts_tablet_manager.cc:1428] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:46:53.572067 15734 heartbeater.cc:503] Master 127.2.182.62:43393 was elected leader, sending a full tablet report...
I20260924 10:46:53.573274 16040 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Bootstrap starting.
I20260924 10:46:53.575722 16040 tablet_bootstrap.cc:654] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:53.576629 16040 log.cc:824] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:53.579243 16040 tablet_bootstrap.cc:492] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: No bootstrap required, opened a new log
I20260924 10:46:53.579501 16040 ts_tablet_manager.cc:1397] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:46:53.583424 16040 raft_consensus.cc:348] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.583760 16040 raft_consensus.cc:374] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:53.583854 16040 raft_consensus.cc:729] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5503b5ee2214fb0b98269437099c965, State: Initialized, Role: FOLLOWER
I20260924 10:46:53.584405 16040 consensus_queue.cc:260] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.585489 16040 ts_tablet_manager.cc:1428] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:46:53.585852 16002 heartbeater.cc:503] Master 127.2.182.62:43393 was elected leader, sending a full tablet report...
W20260924 10:46:53.672281 15735 tablet.cc:2369] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:46:53.686328 16003 tablet.cc:2369] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:46:53.688097 15869 tablet.cc:2369] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:46:53.716360 16045 raft_consensus.cc:482] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:46:53.716624 16045 raft_consensus.cc:504] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.718061 16045 leader_election.cc:288] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 209826dee5704954af2aa96c5198fbaa (127.2.182.1:40503), 1aa4698e54614eb28f72742e0b459056 (127.2.182.2:42011)
I20260924 10:46:53.723345 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.718484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:58215 (local address 127.2.182.2:42011)
0924 10:46:53.718648 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.718654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.718926 (+ 272us) server_negotiation.cc:407] Connection header received
0924 10:46:53.719097 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.719101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.719153 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.719193 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:53.720049 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.720921 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.722378 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.722771 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.722885 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.722946 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.723122 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.723190 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:46:53.723363 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.718299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42011 (local address 127.2.182.3:58215)
0924 10:46:53.718811 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.718826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:53.718952 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:53.719355 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:53.719358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:53.719388 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:53.719529 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.719536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.721028 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.721032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.722248 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.722258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.722890 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.722893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.722995 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.723014 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:46:53.723106 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:46:53.723175 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":276,"thread_start_us":109,"threads_started":1}
I20260924 10:46:53.724054 15813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5135ecbfaa9a43e59431f880c01c4e2c" candidate_uuid: "a5503b5ee2214fb0b98269437099c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa4698e54614eb28f72742e0b459056" is_pre_election: true
I20260924 10:46:53.724390 15813 raft_consensus.cc:2454] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5503b5ee2214fb0b98269437099c965 in term 0.
I20260924 10:46:53.724553 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.718300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40503 (local address 127.2.182.3:47897)
0924 10:46:53.718811 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.718826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:53.718981 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:53.719285 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:53.719289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:53.719312 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:46:53.719529 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.719536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.721975 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.721979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.723153 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:53.723164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.724099 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.724102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:53.724201 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.724224 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:46:53.724337 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:46:53.724404 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":21,"spinlock_wait_cycles":50688,"thread_start_us":122,"threads_started":1}
I20260924 10:46:53.724623 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.718512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:47897 (local address 127.2.182.1:40503)
0924 10:46:53.718637 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.718641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.718973 (+ 332us) server_negotiation.cc:407] Connection header received
0924 10:46:53.719095 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.719099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.719154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.719250 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:46:53.721028 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.721848 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.723322 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.723961 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.724075 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.724131 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.724383 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.724467 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:46:53.724910 15889 leader_election.cc:302] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa4698e54614eb28f72742e0b459056, a5503b5ee2214fb0b98269437099c965; no voters:
I20260924 10:46:53.725208 16045 raft_consensus.cc:2781] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:46:53.725211 15688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5135ecbfaa9a43e59431f880c01c4e2c" candidate_uuid: "a5503b5ee2214fb0b98269437099c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209826dee5704954af2aa96c5198fbaa" is_pre_election: true
I20260924 10:46:53.725294 16045 raft_consensus.cc:482] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:46:53.725345 16045 raft_consensus.cc:3037] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:53.725472 15688 raft_consensus.cc:2454] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5503b5ee2214fb0b98269437099c965 in term 0.
I20260924 10:46:53.726517 16045 raft_consensus.cc:504] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.726888 16045 leader_election.cc:288] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [CANDIDATE]: Term 1 election: Requested vote from peers 209826dee5704954af2aa96c5198fbaa (127.2.182.1:40503), 1aa4698e54614eb28f72742e0b459056 (127.2.182.2:42011)
I20260924 10:46:53.727170 15688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5135ecbfaa9a43e59431f880c01c4e2c" candidate_uuid: "a5503b5ee2214fb0b98269437099c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209826dee5704954af2aa96c5198fbaa"
I20260924 10:46:53.727252 15813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5135ecbfaa9a43e59431f880c01c4e2c" candidate_uuid: "a5503b5ee2214fb0b98269437099c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa4698e54614eb28f72742e0b459056"
I20260924 10:46:53.727337 15688 raft_consensus.cc:3037] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:53.727391 15813 raft_consensus.cc:3037] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:53.728432 15813 raft_consensus.cc:2454] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5503b5ee2214fb0b98269437099c965 in term 1.
I20260924 10:46:53.728663 15688 raft_consensus.cc:2454] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5503b5ee2214fb0b98269437099c965 in term 1.
I20260924 10:46:53.728763 15889 leader_election.cc:302] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa4698e54614eb28f72742e0b459056, a5503b5ee2214fb0b98269437099c965; no voters:
I20260924 10:46:53.728981 16045 raft_consensus.cc:2781] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:53.729238 16045 raft_consensus.cc:686] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 1 LEADER]: Becoming Leader. State: Replica: a5503b5ee2214fb0b98269437099c965, State: Running, Role: LEADER
I20260924 10:46:53.729598 16045 consensus_queue.cc:237] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } }
I20260924 10:46:53.733355 15535 catalog_manager.cc:5495] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5503b5ee2214fb0b98269437099c965 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "a5503b5ee2214fb0b98269437099c965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5503b5ee2214fb0b98269437099c965" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 } health_report { overall_health: UNKNOWN } } }
I20260924 10:46:53.751223 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.747007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.2.182.3:44365)
0924 10:46:53.747157 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:53.747161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:53.747177 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:46:53.747301 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:53.747304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:53.747372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:53.747487 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:46:53.747747 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.748517 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.749474 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:53.749817 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:53.749944 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:53.750222 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:46:53.750757 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:46:53.750875 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:53.751000 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:46:53.751069 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:46:53.757193 15813 raft_consensus.cc:1261] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 1 FOLLOWER]: Refusing update from remote peer a5503b5ee2214fb0b98269437099c965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:53.757195 15688 raft_consensus.cc:1261] T 5135ecbfaa9a43e59431f880c01c4e2c P 209826dee5704954af2aa96c5198fbaa [term 1 FOLLOWER]: Refusing update from remote peer a5503b5ee2214fb0b98269437099c965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:53.757916 16045 consensus_queue.cc:1035] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aa4698e54614eb28f72742e0b459056" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:53.758157 16055 consensus_queue.cc:1035] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209826dee5704954af2aa96c5198fbaa" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:53.765743 16058 mvcc.cc:205] Tried to move back new op lower bound from 7332850949142032384 to 7332850949039247360. Current Snapshot: MvccSnapshot[applied={T|T < 7332850949142032384}]
I20260924 10:46:53.767652 16060 mvcc.cc:205] Tried to move back new op lower bound from 7332850949142032384 to 7332850949039247360. Current Snapshot: MvccSnapshot[applied={T|T < 7332850949142032384}]
I20260924 10:46:53.769124 16059 mvcc.cc:205] Tried to move back new op lower bound from 7332850949142032384 to 7332850949039247360. Current Snapshot: MvccSnapshot[applied={T|T < 7332850949142032384}]
I20260924 10:46:53.775882 15535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:46:53.776044 15535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:46:53.778599 15535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9507338246d848d792e4a53dddd34861: Sending DeleteTablet for 3 replicas of tablet 5135ecbfaa9a43e59431f880c01c4e2c
I20260924 10:46:53.779407 15668 tablet_service.cc:1514] Processing DeleteTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:53 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:60556
I20260924 10:46:53.779742 15936 tablet_service.cc:1514] Processing DeleteTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:53 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:51370
I20260924 10:46:53.779909 15802 tablet_service.cc:1514] Processing DeleteTablet for tablet 5135ecbfaa9a43e59431f880c01c4e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:46:53 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:34450
I20260924 10:46:53.780313 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15574
I20260924 10:46:53.780427 16068 tablet_replica.cc:330] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: stopping tablet replica
I20260924 10:46:53.780705 16068 raft_consensus.cc:2229] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:46:53.781075 16068 raft_consensus.cc:2258] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:46:53.785127 16069 tablet_replica.cc:330] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: stopping tablet replica
I20260924 10:46:53.785507 16069 raft_consensus.cc:2229] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:46:53.785872 16069 raft_consensus.cc:2258] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:46:53.787559 16069 ts_tablet_manager.cc:1905] T 5135ecbfaa9a43e59431f880c01c4e2c P 1aa4698e54614eb28f72742e0b459056: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:46:53.787901 15523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:40503 (error 108)
I20260924 10:46:53.788123 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15738
I20260924 10:46:53.790637 16068 ts_tablet_manager.cc:1905] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:46:53.793426 16068 ts_tablet_manager.cc:1918] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:46:53.793574 16068 log.cc:1196] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/5135ecbfaa9a43e59431f880c01c4e2c
I20260924 10:46:53.793581 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.793226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:40503 (local address 127.0.0.1:60562)
0924 10:46:53.793415 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.793487 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 10:46:53.793833 15523 catalog_manager.cc:4584] TS 209826dee5704954af2aa96c5198fbaa (127.2.182.1:40503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5135ecbfaa9a43e59431f880c01c4e2c: Network error: Client connection negotiation failed: client connection to 127.2.182.1:40503: connect: Connection refused (error 111)
I20260924 10:46:53.793947 16068 ts_tablet_manager.cc:1939] T 5135ecbfaa9a43e59431f880c01c4e2c P a5503b5ee2214fb0b98269437099c965: Deleting consensus metadata
I20260924 10:46:53.795094 15520 catalog_manager.cc:4841] TS a5503b5ee2214fb0b98269437099c965 (127.2.182.3:44365): tablet 5135ecbfaa9a43e59431f880c01c4e2c (table test-table [id=479b0c3fe1404358a6ad1b3f11900cf9]) successfully deleted
I20260924 10:46:53.796128 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15871
I20260924 10:46:53.798206 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:53.796235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42011 (local address 127.0.0.1:34452)
0924 10:46:53.798041 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:53.798111 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:42011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1698}
I20260924 10:46:53.802106 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 15503
I20260924 10:46:53.809621 2776 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:46:54.144999 2776 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:46:54.145095 2776 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:46:54.491853 2776 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:46:54.492 UTC [14570] LOG: received smart shutdown request
2026-09-24 10:46:54.495 UTC [14570] LOG: background worker "logical replication launcher" (PID 14579) exited with exit code 1
2026-09-24 10:46:54.496 UTC [14574] LOG: shutting down
2026-09-24 10:46:54.664 UTC [14570] LOG: database system is shut down
2026-09-24T10:46:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47108 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:46:54 dist-test-slave-6lwt krb5kdc[16079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:46:54 dist-test-slave-6lwt krb5kdc[16079](info): set up 2 sockets
Sep 24 10:46:54 dist-test-slave-6lwt krb5kdc[16079](info): commencing operation
krb5kdc: starting...
W20260924 10:46:55.817189 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:46:55 dist-test-slave-6lwt krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:46:55.935961 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46005
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:56.044569 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:56.044890 16092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:56.044993 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:56.049023 16092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:56.049122 16092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:56.049160 16092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:56.049196 16092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:56.049245 16092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:56.054638 16092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:56.055742 16092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:56.056788 16092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:56.064931 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:56.064931 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:56.065119 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:56.065485 16092 server_base.cc:1034] running on GCE node
I20260924 10:46:56.066015 16092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:56.067154 16092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:56.068311 16092 hybrid_clock.cc:648] HybridClock initialized: now 1790246816068283 us; error 43 us; skew 500 ppm
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:56.070996 16092 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:46:56.071930 16092 webserver.cc:460] Webserver started at http://127.2.182.62:41893/ using document root <none> and password file <none>
I20260924 10:46:56.072494 16092 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:56.072583 16092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:56.072819 16092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:46:56.074838 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "866a887fc16f4e188efe5c80237b1394"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "d75c6a6b902b37a1e1afb061719919b7"
server_key_iv: "5521d57987d78154a9bf4db24cdaa27b"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.075404 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "866a887fc16f4e188efe5c80237b1394"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "d75c6a6b902b37a1e1afb061719919b7"
server_key_iv: "5521d57987d78154a9bf4db24cdaa27b"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.079085 16092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:46:56.081475 16107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:56.082688 16092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:46:56.082826 16092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "866a887fc16f4e188efe5c80237b1394"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "d75c6a6b902b37a1e1afb061719919b7"
server_key_iv: "5521d57987d78154a9bf4db24cdaa27b"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.082939 16092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:56.101225 16092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:56.104562 16092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:56.104758 16092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:56.111507 16092 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:46005
I20260924 10:46:56.111542 16159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:46005 every 8 connection(s)
I20260924 10:46:56.112454 16092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:46:56.115322 16160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:56.119205 16160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.121845 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16092
I20260924 10:46:56.121963 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:46:56.122262 2776 external_mini_cluster.cc:1428] Setting key fd764041ba011d8bcb859a4b5bb3339d
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.61@KRBTEST.COM" created.
I20260924 10:46:56.135998 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.121499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.2.182.62:46005)
0924 10:46:56.122052 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.122064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.122269 (+ 205us) server_negotiation.cc:407] Connection header received
0924 10:46:56.123208 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.123217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.123476 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.123688 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.124845 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.126323 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.127388 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.128059 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.128275 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.131048 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.131054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.131064 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.131085 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.132762 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.133318 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.133322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.133323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.133426 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.133842 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.133844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.133846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.134347 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.134496 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.134875 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.135107 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":4352,"thread_start_us":165,"threads_started":1}
I20260924 10:46:56.136003 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.121499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46005 (local address 127.0.0.1:54590)
0924 10:46:56.121986 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.122041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.122894 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.123778 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.123785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.124269 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.124657 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.124679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.126479 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.126484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.127258 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.127267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.128209 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.128211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.128336 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.128991 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.129007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.130900 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.132885 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.132897 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.132931 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.133218 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.133544 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.133547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.133549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.133743 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.134478 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.134483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.134606 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.134750 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.135005 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":3,"thread_start_us":154,"threads_started":1}
I20260924 10:46:56.138610 16160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.139358 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.139049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52862)
0924 10:46:56.139187 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.139273 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:46:56.139559 16108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
I20260924 10:46:56.140364 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.140116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52868)
0924 10:46:56.140236 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.140296 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:46:56.140508 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:46105: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
I20260924 10:46:56.149384 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:46105
--webserver_interface=127.2.182.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:46:56.189777 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } attempt: 1
I20260924 10:46:56.190748 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.190378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52876)
0924 10:46:56.190565 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.190641 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:46:56.191468 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.191261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52884)
0924 10:46:56.191361 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.191411 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:46:56.191653 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:46105: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
W20260924 10:46:56.257472 16167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:56.257723 16167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:56.257787 16167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:46:56.259868 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } attempt: 2
I20260924 10:46:56.260740 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.260403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52900)
0924 10:46:56.260573 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.260644 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:46:56.261586 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.261367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52910)
0924 10:46:56.261472 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.261522 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:46:56.261729 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:46105: Network error: Client connection negotiation failed: client connection to 127.2.182.61:46105: connect: Connection refused (error 111)
W20260924 10:46:56.262271 16167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:56.262315 16167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:56.262334 16167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:56.262351 16167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:56.262372 16167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:56.267544 16167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:56.268491 16167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:56.269500 16167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:56.277667 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:56.277735 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:56.277724 16167 server_base.cc:1034] running on GCE node
W20260924 10:46:56.277673 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:56.278431 16167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:56.279479 16167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:56.280663 16167 hybrid_clock.cc:648] HybridClock initialized: now 1790246816280638 us; error 48 us; skew 500 ppm
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:56.287174 16167 init.cc:377] Logged in from keytab as kudu/127.2.182.61@KRBTEST.COM (short username kudu)
I20260924 10:46:56.288054 16167 webserver.cc:460] Webserver started at http://127.2.182.61:36559/ using document root <none> and password file <none>
I20260924 10:46:56.288609 16167 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:56.288709 16167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:56.288990 16167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:46:56.290992 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data/instance:
uuid: "b80f0c52e2d54151ad83f8f97997dfd4"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5889dc4ce93564134e9cb392b4af7df7"
server_key_iv: "bfebe1f7194e04d61f74d0b6270b8c45"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.291534 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal/instance:
uuid: "b80f0c52e2d54151ad83f8f97997dfd4"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5889dc4ce93564134e9cb392b4af7df7"
server_key_iv: "bfebe1f7194e04d61f74d0b6270b8c45"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.295143 16167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:46:56.297497 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:56.298801 16167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:46:56.298936 16167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "b80f0c52e2d54151ad83f8f97997dfd4"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5889dc4ce93564134e9cb392b4af7df7"
server_key_iv: "bfebe1f7194e04d61f74d0b6270b8c45"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.299042 16167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:56.329799 16167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:56.333187 16167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:56.333393 16167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:56.340471 16167 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.61:46105
I20260924 10:46:56.340495 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.61:46105 every 8 connection(s)
I20260924 10:46:56.341480 16167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
I20260924 10:46:56.344172 16235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:56.345178 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16167
I20260924 10:46:56.345304 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-1/wal/instance
I20260924 10:46:56.345582 2776 external_mini_cluster.cc:1428] Setting key 72a3f666c31f4e3964b699b89e8557dd
I20260924 10:46:56.348644 16235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.352882 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } attempt: 3
WARNING: no policy specified for kudu/127.2.182.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.60@KRBTEST.COM" created.
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:56.364324 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.353288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52926)
0924 10:46:56.353393 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.353408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.353478 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.354190 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.354193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.354384 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.354506 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.354515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.355681 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.355684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.356496 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.356506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.357328 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.357335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.357477 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.358001 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.358022 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.359788 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.362248 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.362254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.362259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.362458 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.363216 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.363219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.363221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.363321 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.363858 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.363861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.363956 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.364053 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.364166 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 10:46:56.365000 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.351464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46005 (local address 127.0.0.1:54602)
0924 10:46:56.351934 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.351986 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.352819 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.353624 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.353635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.353981 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.354680 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.354692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.355500 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.355505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.356683 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.356691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.357476 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.357483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.357574 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.358212 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.358228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.360112 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.361909 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.361919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.361929 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.362211 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.362528 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.362534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.362539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.362704 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.363173 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.363178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.363445 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.363662 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.363965 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":6,"spinlock_wait_cycles":15104,"thread_start_us":140,"threads_started":1}
I20260924 10:46:56.365012 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.353409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.2.182.61:46105)
0924 10:46:56.353722 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.353737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.353759 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:46:56.353805 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.353811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.354007 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.354184 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.354649 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.355580 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.356606 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.357198 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.357355 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.359926 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.359933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.359951 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.359972 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.362129 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.362594 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.362599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.362601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.362692 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.363421 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.363424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.363426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.363688 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.363811 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.364703 (+ 892us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.364844 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
I20260924 10:46:56.364706 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.351067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.2.182.62:46005)
0924 10:46:56.351333 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.351340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.352106 (+ 766us) server_negotiation.cc:407] Connection header received
0924 10:46:56.353122 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.353126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.353191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.353316 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.354817 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.355392 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.356822 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.357197 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.357311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.360310 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.360314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.360318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.360347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.361795 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.362311 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.362316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.362319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.362419 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.362806 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.362812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.362814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.363064 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.363176 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.364406 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.364542 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 10:46:56.366909 16235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.366995 16160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.367743 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.367420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44820)
0924 10:46:56.367588 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.367643 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:46:56.368577 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.368301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44822)
0924 10:46:56.368464 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.368515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 10:46:56.368803 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
I20260924 10:46:56.376067 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:45853
--webserver_interface=127.2.182.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:46:56.376282 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.367950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52932)
0924 10:46:56.368151 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.368165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.368251 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.368893 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.368898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.369083 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.369186 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.369192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.370171 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.370175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.370919 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.370928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.371401 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.371405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.371495 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.371962 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.371975 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.373272 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.374799 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.374802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.374804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.374982 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.375323 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.375328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.375330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.375418 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.375800 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.375803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.375881 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.375989 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.376118 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 10:46:56.376307 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.368498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.2.182.61:46105)
0924 10:46:56.368599 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.368603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.368617 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:46:56.368655 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.368657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.368697 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.368782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.369300 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.370074 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.371026 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.371274 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.371403 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.373400 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.373403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.373406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.373426 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.374629 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.375073 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.375094 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.375096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.375195 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.375514 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.375520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.375525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.375702 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.375811 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.376041 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.376173 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":42}
I20260924 10:46:56.377300 16235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.377993 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.377705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44828)
0924 10:46:56.377824 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.377909 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:46:56.378499 16186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
I20260924 10:46:56.379307 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.379112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44842)
0924 10:46:56.379212 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.379254 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:46:56.379477 16235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
I20260924 10:46:56.399976 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 1
I20260924 10:46:56.400870 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.400521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44852)
0924 10:46:56.400691 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.400772 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:46:56.401574 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.401366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44856)
0924 10:46:56.401467 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.401516 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:46:56.401721 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
I20260924 10:46:56.428743 16235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 1
I20260924 10:46:56.429689 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.429340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44860)
0924 10:46:56.429516 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.429593 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:46:56.430526 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.430295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44868)
0924 10:46:56.430411 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.430465 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:46:56.430673 16235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
I20260924 10:46:56.476972 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 2
I20260924 10:46:56.477835 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.477489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44884)
0924 10:46:56.477660 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.477739 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:46:56.478590 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.478352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44898)
0924 10:46:56.478464 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.478518 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:46:56.478752 16160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
W20260924 10:46:56.488741 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:56.489102 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:56.489205 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:56.493343 16241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:56.493415 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:56.493460 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:56.493502 16241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:56.493553 16241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:56.498853 16241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46005,127.2.182.61:46105,127.2.182.60:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:56.498885 16235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 2
I20260924 10:46:56.499800 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.499473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44914)
0924 10:46:56.499645 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.499718 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:46:56.500275 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:56.500595 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.500356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44926)
0924 10:46:56.500472 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.500531 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:46:56.500735 16235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:45853: Network error: Client connection negotiation failed: client connection to 127.2.182.60:45853: connect: Connection refused (error 111)
I20260924 10:46:56.501490 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:56.510083 16248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:56.510084 16250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:56.510107 16247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:56.510375 16241 server_base.cc:1034] running on GCE node
I20260924 10:46:56.511112 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:56.512202 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:56.513393 16241 hybrid_clock.cc:648] HybridClock initialized: now 1790246816513380 us; error 27 us; skew 500 ppm
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:56.516170 16241 init.cc:377] Logged in from keytab as kudu/127.2.182.60@KRBTEST.COM (short username kudu)
I20260924 10:46:56.517129 16241 webserver.cc:460] Webserver started at http://127.2.182.60:44087/ using document root <none> and password file <none>
I20260924 10:46:56.517690 16241 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:56.517781 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:56.518015 16241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:46:56.520040 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data/instance:
uuid: "0dccd57e14674957862b6b0790209439"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5b59b12ab91b6512138ba150a83426c2"
server_key_iv: "178214f44ea5d0270755cef2392535fa"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.520612 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal/instance:
uuid: "0dccd57e14674957862b6b0790209439"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5b59b12ab91b6512138ba150a83426c2"
server_key_iv: "178214f44ea5d0270755cef2392535fa"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.524259 16241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 10:46:56.526659 16257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:56.527843 16241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:46:56.527978 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "0dccd57e14674957862b6b0790209439"
format_stamp: "Formatted at 2026-09-24 10:46:56 on dist-test-slave-6lwt"
server_key: "5b59b12ab91b6512138ba150a83426c2"
server_key_iv: "178214f44ea5d0270755cef2392535fa"
server_key_version: "encryptionkey@0"
I20260924 10:46:56.528085 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:56.546998 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:56.549908 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:56.550101 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:56.556665 16241 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.60:45853
I20260924 10:46:56.556749 16309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.60:45853 every 8 connection(s)
I20260924 10:46:56.557864 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
I20260924 10:46:56.560532 16310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:56.562949 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16241
I20260924 10:46:56.563081 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246662595599-2776-0/minicluster-data/master-2/wal/instance
I20260924 10:46:56.563396 2776 external_mini_cluster.cc:1428] Setting key 71739b0093314f3839a18b7a821e0ce8
I20260924 10:46:56.564643 16310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:46:56.575911 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.566422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.2.182.62:46005)
0924 10:46:56.566552 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.566557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.567157 (+ 600us) server_negotiation.cc:407] Connection header received
0924 10:46:56.567325 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.567328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.567377 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.567460 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.568135 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.568774 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.569813 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.570054 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.570164 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.572340 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.572345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.572350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.572379 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.573903 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.574416 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.574420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.574421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.574501 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.574824 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.574829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.574831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.575010 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.575108 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.575597 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.575742 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:46:56.576747 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.566341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46005 (local address 127.0.0.1:54606)
0924 10:46:56.567040 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.567074 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.567213 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.567508 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.567513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.567850 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.567999 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.568008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.568957 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.568960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.569689 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.569699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.570195 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.570200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.570326 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.570784 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.570799 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.572191 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.574011 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.574021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.574031 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.574308 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.574590 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.574595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.574597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.574728 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.575119 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.575126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.575361 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.575544 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.575841 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":96,"threads_started":1}
I20260924 10:46:56.576913 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.564757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.2.182.60:45853)
0924 10:46:56.565222 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.565243 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.565276 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:46:56.565892 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.565908 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.566213 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.566565 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.567381 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.568330 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.569372 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.569823 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.570004 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.572908 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.572916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.572928 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.572951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.574619 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.575053 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.575063 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.575065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.575134 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.575412 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.575416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.575418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.575633 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.575840 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.576528 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.576655 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260924 10:46:56.577899 16160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 3
I20260924 10:46:56.578577 16310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:56.587409 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.578301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44950)
0924 10:46:56.578429 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.578443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.578515 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.579004 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.579007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.579193 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.579304 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.579311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.580316 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.580322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.581096 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.581107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.581543 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.581546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.581627 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.582155 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.582169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.583630 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.585780 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.585784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.585787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.586030 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.586342 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.586345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.586347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.586433 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.586823 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.586826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.586926 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.587043 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.587178 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":18}
I20260924 10:46:56.587597 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.578614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.2.182.60:45853)
0924 10:46:56.578751 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.578755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.578767 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:46:56.578802 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.578804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.578835 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.578971 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.579434 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.580168 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.581203 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.581427 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.581540 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.583776 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.583779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.583782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.583807 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.585688 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.586130 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.586134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.586137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.586231 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.586526 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.586530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.586531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.586740 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.586836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.587122 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.587236 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:46:56.588703 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.579043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:46105 (local address 127.0.0.1:52948)
0924 10:46:56.579121 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.579134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.579201 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.579615 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.579618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.579789 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.579911 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.579921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.580693 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.580696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.581495 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.581504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.582272 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.582274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.582347 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.583104 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.583114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.584644 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.586828 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.586831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.586833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.587075 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.587706 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.587709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.587711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.587802 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.588231 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.588236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.588326 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.588434 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.588558 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:46:56.588747 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.579298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.2.182.61:46105)
0924 10:46:56.579428 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.579431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.579442 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:46:56.579479 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.579481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.579514 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.579603 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.580057 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.580544 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.581618 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.582043 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.582124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.584823 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.584827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.584830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.584857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.586732 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.587487 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.587491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.587494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.587584 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.587885 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.587892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.587894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.588128 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.588229 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.588502 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.588606 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:46:56.589891 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.579443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.2.182.62:46005)
0924 10:46:56.579536 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.579539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.579684 (+ 145us) server_negotiation.cc:407] Connection header received
0924 10:46:56.579851 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.579854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.579897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.579982 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.580666 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.581649 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.582677 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.582924 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.583002 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.585486 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.585491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.585493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.585517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.588011 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.588497 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.588501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.588504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.588600 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.588943 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.588946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.588948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.589176 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.589381 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.589697 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.589791 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":38}
I20260924 10:46:56.591812 16160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: Bootstrap starting.
I20260924 10:46:56.591900 16235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } attempt: 3
I20260924 10:46:56.593197 16310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:46:56.594749 16160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:56.595623 16160 log.cc:824] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:46:56.598845 16160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: No bootstrap required, opened a new log
I20260924 10:46:56.600651 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.591004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.2.182.61:46105)
0924 10:46:56.591117 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.591121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.591135 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:46:56.591245 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.591247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.591279 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.591339 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.592018 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.592712 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.593780 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.594105 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.594186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.596874 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.596880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.596885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.596937 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.598580 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.599128 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.599135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.599137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.599237 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.599570 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.599573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.599575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.599778 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.599996 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.600375 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.600489 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 10:46:56 dist-test-slave-6lwt krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:46:56.602188 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.592242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44960)
0924 10:46:56.592384 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.592398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.592462 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.592684 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.592686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.592854 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.592968 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.592974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.594217 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.594221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.595299 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.595309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.595896 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.595900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.596002 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.596439 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.596454 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.597989 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.599846 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.599851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.599853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.600104 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.600381 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.600387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.600390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.600484 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.601327 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.601330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.601416 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.601597 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.602020 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 10:46:56.602903 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.592338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.2.182.60:45853)
0924 10:46:56.592430 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.592434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.592446 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:46:56.592534 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.592537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.592571 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.592651 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.593096 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.594090 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.595445 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.595777 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.595877 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.598145 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.598149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.598151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.598176 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.599746 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.600204 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.600207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.600209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.600289 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.600598 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.600601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.600603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.601148 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.601277 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.602616 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.602762 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:46:56.604578 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:46:56.604452 16160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.604679 16160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:56.604740 16160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866a887fc16f4e188efe5c80237b1394, State: Initialized, Role: FOLLOWER
I20260924 10:46:56.605230 16160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.605680 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.605863 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.606020 16235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: Bootstrap starting.
I20260924 10:46:56.606596 16160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:56.608875 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.593624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:45853 (local address 127.0.0.1:44962)
0924 10:46:56.594033 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 10:46:56.594046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:46:56.594114 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:46:56.596449 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:46:56.596451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:46:56.596629 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:46:56.596713 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.596723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.597713 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.597717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.598465 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:46:56.598478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.599080 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.599087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:46:56.599932 (+ 845us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.600426 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:46:56.600442 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:46:56.601833 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:46:56.607361 (+ 5528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.607364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.607366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.607589 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.607904 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:46:56.607907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:46:56.607908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:46:56.607980 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:46:56.608379 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:46:56.608381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:46:56.608451 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 10:46:56.608540 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:46:56.608658 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356}
I20260924 10:46:56.608879 16235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:56.609447 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.593839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.2.182.60:45853)
0924 10:46:56.596130 (+ 2291us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.596142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.596159 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:46:56.596195 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.596199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.596254 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.596338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.596836 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.597588 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.598590 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.598939 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.599051 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.602008 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.602011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.602014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.602037 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.607240 (+ 5203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.607741 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.607744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.607745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.607822 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.608058 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.608060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.608062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.608258 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.608348 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.609190 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.609296 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2228,"thread_start_us":95,"threads_started":1}
I20260924 10:46:56.610062 16235 log.cc:824] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: Log is configured to *not* fsync() on all Append() calls
W20260924 10:46:56.611830 16332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:46:56.611953 16332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:46:56.612226 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: Bootstrap starting.
I20260924 10:46:56.613744 16235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: No bootstrap required, opened a new log
I20260924 10:46:56.618016 16235 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.618266 16235 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:56.618324 16235 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80f0c52e2d54151ad83f8f97997dfd4, State: Initialized, Role: FOLLOWER
I20260924 10:46:56.618814 16235 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.619918 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.610007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.2.182.62:46005)
0924 10:46:56.610115 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.610119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.610653 (+ 534us) server_negotiation.cc:407] Connection header received
0924 10:46:56.610843 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.610846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.610896 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.610954 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.611916 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.612625 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.613865 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.614191 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.614283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.615983 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.615990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.615993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.616018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.617830 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.618273 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.618277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.618279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.618357 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.618628 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.618631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.618632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.618856 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.618989 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.619632 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.619763 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:46:56.620098 16235 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:56.616055 16092 master_runner.cc:186] Error getting master registration for 127.2.182.61:46105: OK, instance_id { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" instance_seqno: 1790246816338361 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:46:56.621140 16092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:46:56.621256 16310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:56.622192 16310 log.cc:824] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: Log is configured to *not* fsync() on all Append() calls
W20260924 10:46:56.626312 16349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:46:56.626391 16349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:46:56.627378 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.611300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.2.182.61:46105)
0924 10:46:56.611415 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.611419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.616699 (+ 5280us) server_negotiation.cc:407] Connection header received
0924 10:46:56.616915 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.616921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.616972 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.617044 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.617776 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.618521 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.619801 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.620121 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.620197 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.622028 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.622031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.622034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.622057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.623705 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.625125 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.625128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.625130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.625214 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.626347 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.626351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.626353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.626587 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.626709 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.627089 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.627211 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:46:56.628520 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: No bootstrap required, opened a new log
I20260924 10:46:56.627720 16167 master_runner.cc:186] Error getting master registration for 127.2.182.60:45853: OK, instance_id { permanent_uuid: "0dccd57e14674957862b6b0790209439" instance_seqno: 1790246816554862 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:46:56.630950 16167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:46:56.632227 16310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.632400 16310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:56.632480 16310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dccd57e14674957862b6b0790209439, State: Initialized, Role: FOLLOWER
I20260924 10:46:56.633015 16310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.634368 16310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:56.634737 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:56.610641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.2.182.60:45853)
0924 10:46:56.610752 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:56.610758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:56.612526 (+ 1768us) server_negotiation.cc:407] Connection header received
0924 10:46:56.612681 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:56.612687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:56.612733 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:56.612782 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:56.613706 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.614437 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.615578 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:56.615949 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:56.616017 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:56.624168 (+ 8151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:56.624172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:56.624174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:56.624198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:56.625982 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.633378 (+ 7396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.633382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.633384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.633467 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:56.633763 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:56.633766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:56.633767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:56.633968 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:56.634101 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:56.634462 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 10:46:56.634583 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:46:56.637087 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.637293 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.637693 16352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.637816 16352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:46:56.640484 16363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:46:56.640571 16363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:46:56.642014 16241 master_runner.cc:263] No leader master found from master 0dccd57e14674957862b6b0790209439
I20260924 10:46:56.642086 16241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 10:46:56.801457 16263 tablet.cc:2369] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:46:56.835008 16188 tablet.cc:2369] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:46:56.856783 16113 tablet.cc:2369] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:46:56.878432 16317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:46:56.878626 16317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.879568 16317 leader_election.cc:288] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b80f0c52e2d54151ad83f8f97997dfd4 (127.2.182.61:46105), 0dccd57e14674957862b6b0790209439 (127.2.182.60:45853)
I20260924 10:46:56.880180 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "866a887fc16f4e188efe5c80237b1394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dccd57e14674957862b6b0790209439" is_pre_election: true
I20260924 10:46:56.880167 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "866a887fc16f4e188efe5c80237b1394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" is_pre_election: true
I20260924 10:46:56.880505 16284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 866a887fc16f4e188efe5c80237b1394 in term 0.
I20260924 10:46:56.880506 16209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 866a887fc16f4e188efe5c80237b1394 in term 0.
I20260924 10:46:56.880970 16111 leader_election.cc:302] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dccd57e14674957862b6b0790209439, 866a887fc16f4e188efe5c80237b1394; no voters:
I20260924 10:46:56.881254 16317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:46:56.881337 16317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:46:56.881381 16317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:56.882447 16317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.882764 16317 leader_election.cc:288] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [CANDIDATE]: Term 1 election: Requested vote from peers b80f0c52e2d54151ad83f8f97997dfd4 (127.2.182.61:46105), 0dccd57e14674957862b6b0790209439 (127.2.182.60:45853)
I20260924 10:46:56.883090 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "866a887fc16f4e188efe5c80237b1394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dccd57e14674957862b6b0790209439"
I20260924 10:46:56.883082 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "866a887fc16f4e188efe5c80237b1394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80f0c52e2d54151ad83f8f97997dfd4"
I20260924 10:46:56.883217 16284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:56.883217 16209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:56.884510 16209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 866a887fc16f4e188efe5c80237b1394 in term 1.
I20260924 10:46:56.884510 16284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 866a887fc16f4e188efe5c80237b1394 in term 1.
I20260924 10:46:56.884800 16111 leader_election.cc:302] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dccd57e14674957862b6b0790209439, 866a887fc16f4e188efe5c80237b1394; no voters:
I20260924 10:46:56.885082 16317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:56.885380 16317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [term 1 LEADER]: Becoming Leader. State: Replica: 866a887fc16f4e188efe5c80237b1394, State: Running, Role: LEADER
I20260924 10:46:56.885757 16317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } }
I20260924 10:46:56.892477 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 866a887fc16f4e188efe5c80237b1394. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.892621 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:56.893072 16367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:46:56.893898 16367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:46:56.898520 16209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [term 1 FOLLOWER]: Refusing update from remote peer 866a887fc16f4e188efe5c80237b1394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:56.898520 16284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [term 1 FOLLOWER]: Refusing update from remote peer 866a887fc16f4e188efe5c80237b1394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:46:56.899240 16365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:56.899420 16317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:46:56.903052 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 866a887fc16f4e188efe5c80237b1394. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.903059 16352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 866a887fc16f4e188efe5c80237b1394. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.903177 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.903193 16352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.905627 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.905756 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:56.906085 16368 mvcc.cc:205] Tried to move back new op lower bound from 7332850962009255936 to 7332850961967038464. Current Snapshot: MvccSnapshot[applied={T|T < 7332850962009255936}]
I20260924 10:46:56.906126 16352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.906253 16352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.906499 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.906594 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:56.906680 16370 mvcc.cc:205] Tried to move back new op lower bound from 7332850962009255936 to 7332850961967038464. Current Snapshot: MvccSnapshot[applied={T|T < 7332850962009255936}]
I20260924 10:46:56.907346 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "866a887fc16f4e188efe5c80237b1394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866a887fc16f4e188efe5c80237b1394" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46005 } } peers { permanent_uuid: "b80f0c52e2d54151ad83f8f97997dfd4" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 46105 } } peers { permanent_uuid: "0dccd57e14674957862b6b0790209439" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 45853 } } }
I20260924 10:46:56.907455 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:46:56.907452 16367 catalog_manager.cc:1347] Generated new cluster ID: 91a88e4a898f413c894bf990797ecbb2
I20260924 10:46:56.907502 16367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:46:56.920956 16367 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:46:56.922174 16367 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:46:56.932395 16367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 866a887fc16f4e188efe5c80237b1394: Generated new TSK 0
I20260924 10:46:56.933314 16367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:46:57.627718 16349 catalog_manager.cc:1259] Loaded cluster ID: 91a88e4a898f413c894bf990797ecbb2
I20260924 10:46:57.627810 16349 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: loading cluster ID for follower catalog manager: success
I20260924 10:46:57.629561 16349 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: acquiring CA information for follower catalog manager: success
I20260924 10:46:57.630720 16349 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b80f0c52e2d54151ad83f8f97997dfd4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:46:57.641822 16363 catalog_manager.cc:1259] Loaded cluster ID: 91a88e4a898f413c894bf990797ecbb2
I20260924 10:46:57.641896 16363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: loading cluster ID for follower catalog manager: success
I20260924 10:46:57.643543 16363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: acquiring CA information for follower catalog manager: success
I20260924 10:46:57.644716 16363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0dccd57e14674957862b6b0790209439: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:46:57.987615 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16092
I20260924 10:46:57.993930 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16167
I20260924 10:46:58.000229 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16241
2026-09-24T10:46:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3213 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:46:58 dist-test-slave-6lwt krb5kdc[16383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:46:58 dist-test-slave-6lwt krb5kdc[16383](info): set up 2 sockets
Sep 24 10:46:58 dist-test-slave-6lwt krb5kdc[16383](info): commencing operation
krb5kdc: starting...
W20260924 10:46:59.045495 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:46:59.163780 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:39093
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:46:59.271212 16396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:46:59.271440 16396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:46:59.271477 16396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:46:59.275378 16396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:46:59.275426 16396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:46:59.275446 16396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:46:59.275462 16396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:46:59.275477 16396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:46:59.280445 16396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:46:59.281379 16396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:46:59.282248 16396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:46:59.290431 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:59.290505 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:46:59.290449 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:46:59.290452 16396 server_base.cc:1034] running on GCE node
I20260924 10:46:59.291234 16396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:46:59.292150 16396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:46:59.293370 16396 hybrid_clock.cc:648] HybridClock initialized: now 1790246819293341 us; error 39 us; skew 500 ppm
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:46:59.296022 16396 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:46:59.296836 16396 webserver.cc:460] Webserver started at http://127.2.182.62:44041/ using document root <none> and password file <none>
I20260924 10:46:59.297385 16396 fs_manager.cc:359] Metadata directory not provided
I20260924 10:46:59.297473 16396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:46:59.297693 16396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:46:59.299566 16396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "74653db7ca7a41809d69d6ce55b63284"
format_stamp: "Formatted at 2026-09-24 10:46:59 on dist-test-slave-6lwt"
server_key: "2ce8221a972733e8082a2e2246316af8"
server_key_iv: "d6b415c8adef84929518a04e8ad995d1"
server_key_version: "encryptionkey@0"
I20260924 10:46:59.300104 16396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "74653db7ca7a41809d69d6ce55b63284"
format_stamp: "Formatted at 2026-09-24 10:46:59 on dist-test-slave-6lwt"
server_key: "2ce8221a972733e8082a2e2246316af8"
server_key_iv: "d6b415c8adef84929518a04e8ad995d1"
server_key_version: "encryptionkey@0"
I20260924 10:46:59.303537 16396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:46:59.305751 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:46:59.306833 16396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:46:59.306962 16396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "74653db7ca7a41809d69d6ce55b63284"
format_stamp: "Formatted at 2026-09-24 10:46:59 on dist-test-slave-6lwt"
server_key: "2ce8221a972733e8082a2e2246316af8"
server_key_iv: "d6b415c8adef84929518a04e8ad995d1"
server_key_version: "encryptionkey@0"
I20260924 10:46:59.307065 16396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:46:59.341360 16396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:46:59.344826 16396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:46:59.345048 16396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:46:59.352203 16396 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:39093
I20260924 10:46:59.352227 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:39093 every 8 connection(s)
I20260924 10:46:59.353219 16396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:46:59.355844 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:46:59.359578 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16396
I20260924 10:46:59.359710 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:46:59.360088 2776 external_mini_cluster.cc:1428] Setting key 06c20830bd0d19c2220004086c1b40d2
I20260924 10:46:59.362058 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284: Bootstrap starting.
I20260924 10:46:59.364284 16464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284: Neither blocks nor log segments found. Creating new log.
I20260924 10:46:59.365082 16464 log.cc:824] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284: Log is configured to *not* fsync() on all Append() calls
I20260924 10:46:59.366842 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284: No bootstrap required, opened a new log
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:46:59.370759 16464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } }
I20260924 10:46:59.370961 16464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:46:59.371032 16464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74653db7ca7a41809d69d6ce55b63284, State: Initialized, Role: FOLLOWER
I20260924 10:46:59.371502 16464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } }
I20260924 10:46:59.371623 16464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:46:59.371732 16464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:46:59.371852 16464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:46:59.372685 16464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } }
I20260924 10:46:59.373056 16464 leader_election.cc:302] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74653db7ca7a41809d69d6ce55b63284; no voters:
I20260924 10:46:59.373376 16464 leader_election.cc:288] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:46:59.373436 16469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:46:59.373653 16469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [term 1 LEADER]: Becoming Leader. State: Replica: 74653db7ca7a41809d69d6ce55b63284, State: Running, Role: LEADER
I20260924 10:46:59.373987 16469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } }
I20260924 10:46:59.374639 16464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:46:59.374645 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:46:59.361583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.2.182.62:39093)
0924 10:46:59.362054 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 10:46:59.362069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:46:59.362103 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:46:59.362774 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:46:59.362791 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:46:59.363102 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:46:59.363483 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:46:59.364310 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:59.365472 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:59.366309 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:46:59.366776 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:46:59.366943 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:46:59.369693 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:46:59.369698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:46:59.369710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:46:59.369745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:46:59.371583 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:59.372108 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:59.372114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:59.372119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:59.372234 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:46:59.372530 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:46:59.372533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:46:59.372535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:46:59.373005 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:46:59.373191 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:46:59.373453 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:46:59.373770 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":133,"threads_started":1}
I20260924 10:46:59.375538 16470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74653db7ca7a41809d69d6ce55b63284. Latest consensus state: current_term: 1 leader_uuid: "74653db7ca7a41809d69d6ce55b63284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } } }
I20260924 10:46:59.375927 16470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:59.375833 16471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74653db7ca7a41809d69d6ce55b63284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74653db7ca7a41809d69d6ce55b63284" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 39093 } } }
I20260924 10:46:59.376219 16471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284 [sys.catalog]: This master's current role is: LEADER
I20260924 10:46:59.376520 16478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:46:59.377836 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:46:59.380403 16478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:46:59.385732 16478 catalog_manager.cc:1347] Generated new cluster ID: 05e95e6cf05740129c2375f0f7437e4c
I20260924 10:46:59.385804 16478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:46:59.412990 16478 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:46:59.414271 16478 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:46:59.421253 16478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74653db7ca7a41809d69d6ce55b63284: Generated new TSK 0
I20260924 10:46:59.422008 16478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:46:59.433326 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16396
2026-09-24T10:46:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1430 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16491](info): set up 2 sockets
Sep 24 10:46:59 dist-test-slave-6lwt krb5kdc[16491](info): commencing operation
krb5kdc: starting...
W20260924 10:47:00.453492 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:47:00 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:47:00.573694 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:00.681555 16504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:00.681857 16504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:00.681931 16504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:00.685915 16504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:47:00.686009 16504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:00.686048 16504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:00.686107 16504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:47:00.686148 16504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:47:00.691388 16504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:00.692514 16504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:00.693536 16504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:00.701670 16510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:00.701758 16509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:00.701643 16512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:00.701778 16504 server_base.cc:1034] running on GCE node
I20260924 10:47:00.702467 16504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:00.703526 16504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:00.704690 16504 hybrid_clock.cc:648] HybridClock initialized: now 1790246820704676 us; error 26 us; skew 500 ppm
Sep 24 10:47:00 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:00.707392 16504 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:47:00.708261 16504 webserver.cc:460] Webserver started at http://127.2.182.62:46167/ using document root <none> and password file <none>
I20260924 10:47:00.708806 16504 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:00.708894 16504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:00.709149 16504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:00.711210 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "6cb719dd7b6a4b76b99e6018f62b1e18"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "52ba75d086600a94172e2bfbdbc429d8"
server_key_iv: "7879d9f4978df470f975cc8fa4018b85"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.711813 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "6cb719dd7b6a4b76b99e6018f62b1e18"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "52ba75d086600a94172e2bfbdbc429d8"
server_key_iv: "7879d9f4978df470f975cc8fa4018b85"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.715363 16504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:47:00.717661 16519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:00.718825 16504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:00.718969 16504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "6cb719dd7b6a4b76b99e6018f62b1e18"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "52ba75d086600a94172e2bfbdbc429d8"
server_key_iv: "7879d9f4978df470f975cc8fa4018b85"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.719077 16504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:00.735956 16504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:00.739216 16504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:00.739418 16504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:00.746587 16504 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32915
I20260924 10:47:00.746603 16571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32915 every 8 connection(s)
I20260924 10:47:00.747514 16504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:47:00.749848 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16504
I20260924 10:47:00.749977 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:47:00.750303 2776 external_mini_cluster.cc:1428] Setting key 78905ffaac4a20be3d0401d1f1ee03f2
I20260924 10:47:00.750523 16572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:00.756569 16572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap starting.
Sep 24 10:47:00 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:00.759320 16572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:00.760035 16572 log.cc:824] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:00.762135 16572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: No bootstrap required, opened a new log
I20260924 10:47:00.764634 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:00.751692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.2.182.62:32915)
0924 10:47:00.752101 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:00.752115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:00.752148 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:47:00.752806 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:00.752822 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:00.753192 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:00.753512 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:00.754304 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:00.755309 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:00.756165 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:00.756637 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:00.756808 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:00.759482 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:00.759487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:00.759499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:00.759525 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:00.761591 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:00.762015 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:00.762020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:00.762025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:00.762151 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:00.762453 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:00.762458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:00.762461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:00.762948 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:00.763138 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:00.763342 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:47:00.763620 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260924 10:47:00.766158 16572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:00.766376 16572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:00.766436 16572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Initialized, Role: FOLLOWER
I20260924 10:47:00.766892 16572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:00.767047 16572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:47:00.767113 16572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:47:00.767210 16572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:00.768092 16572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:00.768428 16572 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb719dd7b6a4b76b99e6018f62b1e18; no voters:
I20260924 10:47:00.768775 16572 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:47:00.768890 16577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:00.769140 16577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Running, Role: LEADER
I20260924 10:47:00.769491 16577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:00.769976 16572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:47:00.771287 16578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:00.771296 16579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb719dd7b6a4b76b99e6018f62b1e18. Latest consensus state: current_term: 1 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:00.771415 16579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:00.771415 16578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:00.772257 16586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:47:00.774876 16586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:47:00.779918 16586 catalog_manager.cc:1347] Generated new cluster ID: fa4d3498e886402eb117dc559ca46e73
I20260924 10:47:00.779984 16586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:47:00.785640 16586 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:47:00.786780 16586 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:47:00.794720 16586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Generated new TSK 0
I20260924 10:47:00.795480 16586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:47:00.829572 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:00.936820 16598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:00.937068 16598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:00.937109 16598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:00.941118 16598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:00.941171 16598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:00.941228 16598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:47:00.946338 16598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:00.947319 16598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:00.948300 16598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:00.957530 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:00.957654 16598 server_base.cc:1034] running on GCE node
W20260924 10:47:00.957517 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:00.957502 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:00.958196 16598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:00.958734 16598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:00.959887 16598 hybrid_clock.cc:648] HybridClock initialized: now 1790246820959858 us; error 43 us; skew 500 ppm
Sep 24 10:47:00 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:00.962700 16598 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:47:00.963511 16598 webserver.cc:460] Webserver started at http://127.2.182.1:44083/ using document root <none> and password file <none>
I20260924 10:47:00.964046 16598 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:00.964134 16598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:00.964362 16598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:00.966421 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "ee0847995a1b450b8e5e80d7961654a1"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "6745f1ae5b092164fe4ca249feae1784"
server_key_iv: "841488585b12a51d10f17c613958c105"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.967003 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "ee0847995a1b450b8e5e80d7961654a1"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "6745f1ae5b092164fe4ca249feae1784"
server_key_iv: "841488585b12a51d10f17c613958c105"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.970638 16598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 10:47:00.973096 16613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:00.974318 16598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:00.974447 16598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "ee0847995a1b450b8e5e80d7961654a1"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "6745f1ae5b092164fe4ca249feae1784"
server_key_iv: "841488585b12a51d10f17c613958c105"
server_key_version: "encryptionkey@0"
I20260924 10:47:00.974556 16598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:00.987102 16598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:00.989822 16598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:00.990020 16598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:00.990604 16598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:00.991401 16598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:00.991470 16598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:00.991554 16598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:00.991603 16598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.000078 16598 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36423
I20260924 10:47:01.000098 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:36423 every 8 connection(s)
I20260924 10:47:01.000963 16598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:47:01.005354 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16598
I20260924 10:47:01.005491 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:47:01.005803 2776 external_mini_cluster.cc:1428] Setting key 4d6fdb8471230b4ed4668863d4843dae
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:47:01.015054 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.002669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:33925 (local address 127.2.182.62:32915)
0924 10:47:01.002813 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.002817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.003781 (+ 964us) server_negotiation.cc:407] Connection header received
0924 10:47:01.004669 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.004672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.004746 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.004846 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:01.006365 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.006927 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.008622 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.008854 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.009012 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.011442 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:01.011445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:01.011447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:01.011469 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:01.013036 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.013544 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.013549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.013551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.013631 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.013981 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.013984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.013986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.014161 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:01.014272 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.014761 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.014884 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:47:01.015863 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.002979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:33925)
0924 10:47:01.003623 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.003661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.004460 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.005036 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.005043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.005566 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:01.006204 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.006236 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.007053 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.007056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.008492 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.008508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.009054 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.009059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.009232 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.009831 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:01.009847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:01.011272 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:01.013158 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.013165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.013179 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.013438 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.013733 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.013737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.013738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.013878 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.014271 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:01.014278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:01.014520 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.014708 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.015022 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":122,"threads_started":1}
I20260924 10:47:01.017127 16727 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:47:01.017426 16727 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:01.017925 16727 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:01.019423 16536 ts_manager.cc:194] Registered new tserver with Master: ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423)
I20260924 10:47:01.020525 16536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:33925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:47:01.032223 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:01.136857 16732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:01.137135 16732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:01.137204 16732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:01.141039 16732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:01.141110 16732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:01.141194 16732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:47:01.146157 16732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:01.147119 16732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:01.148082 16732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:01.159478 16740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:01.159436 16738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:01.159451 16737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:01.160619 16732 server_base.cc:1034] running on GCE node
I20260924 10:47:01.161070 16732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:01.161744 16732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:01.162947 16732 hybrid_clock.cc:648] HybridClock initialized: now 1790246821162931 us; error 35 us; skew 500 ppm
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:01.165598 16732 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:47:01.166412 16732 webserver.cc:460] Webserver started at http://127.2.182.2:34983/ using document root <none> and password file <none>
I20260924 10:47:01.166947 16732 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:01.167037 16732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:01.167275 16732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:01.169261 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "2fcc3192e92e481d910ca1420e571750"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "22500390bab5493051e33e1632b9c1ca"
server_key_iv: "9a5aba3545654451129b7b207fcbbac5"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.169847 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "2fcc3192e92e481d910ca1420e571750"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "22500390bab5493051e33e1632b9c1ca"
server_key_iv: "9a5aba3545654451129b7b207fcbbac5"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.173391 16732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:47:01.175612 16747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.176730 16732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:01.176858 16732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "2fcc3192e92e481d910ca1420e571750"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "22500390bab5493051e33e1632b9c1ca"
server_key_iv: "9a5aba3545654451129b7b207fcbbac5"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.177021 16732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:01.224431 16732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:01.227278 16732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:01.227474 16732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:01.228050 16732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:01.228852 16732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:01.228955 16732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.229048 16732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:01.229102 16732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.237350 16732 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34641
I20260924 10:47:01.237385 16860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34641 every 8 connection(s)
I20260924 10:47:01.238196 16732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:01.248446 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16732
I20260924 10:47:01.248576 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:47:01.248880 2776 external_mini_cluster.cc:1428] Setting key 087a29ba909f631a7bc9143c1893ebe0
I20260924 10:47:01.251219 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.239859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38305 (local address 127.2.182.62:32915)
0924 10:47:01.240012 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.240017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.241016 (+ 999us) server_negotiation.cc:407] Connection header received
0924 10:47:01.241858 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.241861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.241904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.241980 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:01.243056 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.243606 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.244647 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.244872 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.245016 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.247449 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:01.247452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:01.247454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:01.247475 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:01.248878 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.249557 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.249561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.249562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.249656 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.250026 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.250029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.250031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.250256 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:01.250357 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.250990 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.251097 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:47:01.252457 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.240221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:38305)
0924 10:47:01.240829 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.240867 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.241661 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.242127 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.242133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.242457 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:01.242936 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.242948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.243709 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.243711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.244546 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.244555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.245049 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.245051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.245189 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.245798 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:01.245814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:01.247285 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:01.249075 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.249081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.249097 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.249443 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.249795 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.249798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.249799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.249932 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.250372 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:01.250384 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:01.250712 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.250937 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.251333 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":106,"threads_started":1}
I20260924 10:47:01.253727 16861 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
I20260924 10:47:01.253995 16861 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:01.254448 16861 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:01.255463 16536 ts_manager.cc:194] Registered new tserver with Master: 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
I20260924 10:47:01.256330 16536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:38305
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:47:01.278039 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:01.386523 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:01.386909 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:01.386996 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:01.392026 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:01.392087 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:01.392191 16866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:47:01.397567 16866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:01.398576 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:01.399578 16866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:01.409550 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:01.409600 16866 server_base.cc:1034] running on GCE node
W20260924 10:47:01.409902 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:01.410187 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:01.410554 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:01.411182 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:01.412359 16866 hybrid_clock.cc:648] HybridClock initialized: now 1790246821412318 us; error 51 us; skew 500 ppm
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:01.418807 16866 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:47:01.419632 16866 webserver.cc:460] Webserver started at http://127.2.182.3:40743/ using document root <none> and password file <none>
I20260924 10:47:01.420181 16866 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:01.420267 16866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:01.420483 16866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:01.422602 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "70da2e0d20dc40d2860ebc03f4df0193"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "324ed140ffa05834e755059055590c59"
server_key_iv: "3c5f8389f783f0b1251526c0865fd3ff"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.423202 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "70da2e0d20dc40d2860ebc03f4df0193"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "324ed140ffa05834e755059055590c59"
server_key_iv: "3c5f8389f783f0b1251526c0865fd3ff"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.426951 16866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 10:47:01.429327 16881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.430531 16866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:01.430662 16866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "70da2e0d20dc40d2860ebc03f4df0193"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "324ed140ffa05834e755059055590c59"
server_key_iv: "3c5f8389f783f0b1251526c0865fd3ff"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.430778 16866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:01.443087 16866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:01.446310 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:01.446508 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:01.447073 16866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:01.447860 16866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:01.447927 16866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.447995 16866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:01.448036 16866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.457036 16866 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:41799
I20260924 10:47:01.457069 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:41799 every 8 connection(s)
I20260924 10:47:01.457886 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:47:01.464183 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 16866
I20260924 10:47:01.464326 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:47:01.464584 2776 external_mini_cluster.cc:1428] Setting key 1864fb6ad58a721ecd7f2fba7f732673
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:01.471342 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.459606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:50009 (local address 127.2.182.62:32915)
0924 10:47:01.459760 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.459765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.461070 (+ 1305us) server_negotiation.cc:407] Connection header received
0924 10:47:01.461907 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.461909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.461955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.462028 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:01.463268 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.463802 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.465273 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.465600 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.465720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.468107 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:01.468118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:01.468120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:01.468143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:01.469371 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.469890 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.469895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.469897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.469980 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.470315 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.470320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.470321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.470499 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:01.470606 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.471069 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.471185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:47:01.472159 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.460023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:50009)
0924 10:47:01.460866 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.460976 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.461728 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.462235 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.462244 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.462608 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:01.463139 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.463152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.463903 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.463905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.465139 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.465150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.465767 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.465771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.465947 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.466535 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:01.466551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:01.467947 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:01.469516 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.469522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.469542 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.469783 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.470071 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:01.470075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:01.470077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:01.470217 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:01.470604 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:01.470611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:01.470844 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.471030 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.471341 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":524,"threads_started":1}
I20260924 10:47:01.473260 16995 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
I20260924 10:47:01.473484 16995 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:01.473928 16995 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:01.474869 16536 ts_manager.cc:194] Registered new tserver with Master: 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799)
I20260924 10:47:01.475405 16536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:50009
I20260924 10:47:01.478338 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:47:01.486843 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.480142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.2.182.62:32915)
0924 10:47:01.480294 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.480299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.480532 (+ 233us) server_negotiation.cc:407] Connection header received
0924 10:47:01.480620 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.480622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.480658 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.480721 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:01.481454 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.481918 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.482741 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.482936 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.483014 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.484205 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:01.484208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:01.484219 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:01.484239 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:01.485457 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.485809 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.485811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.485812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.485878 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:01.486090 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:01.486091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:01.486093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:01.486286 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:01.486383 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.486567 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.486655 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:47:01.489413 16536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:47:01.492208 16536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:47:01.505695 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.500739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:41799 (local address 127.0.0.1:37414)
0924 10:47:01.501322 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.501338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.501419 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.502050 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.502056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.502075 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.502170 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.502174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.503087 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.503093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.504366 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.504375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.505306 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.505309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.505392 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.505419 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.505502 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.505583 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":102,"thread_start_us":124,"threads_started":1}
I20260924 10:47:01.505782 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.500987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.2.182.3:41799)
0924 10:47:01.501490 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.501503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.501522 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:47:01.501589 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.501592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.501772 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.501929 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.502280 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.502967 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.504489 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.505184 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.505300 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.505431 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.505545 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.505636 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":126,"threads_started":1}
I20260924 10:47:01.506336 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.500154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.0.0.1:36434)
0924 10:47:01.501111 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.501123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.501285 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.501888 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.501890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.501926 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.502007 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.502016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.503404 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.503410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.504781 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.504793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.506048 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.506051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.506114 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.506130 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.506190 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.506243 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":153,"thread_start_us":101,"threads_started":1}
I20260924 10:47:01.506482 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.500316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.2.182.1:36423)
0924 10:47:01.500659 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.500673 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.501203 (+ 530us) server_negotiation.cc:407] Connection header received
0924 10:47:01.501416 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.501420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.501572 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.501695 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.502112 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.503203 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.504947 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.505739 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.505843 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.505983 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.506254 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.506350 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260924 10:47:01.507939 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.499946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.0.0.1:52782)
0924 10:47:01.500982 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.501029 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.501158 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.501734 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.501740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.501761 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.501902 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.501909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.503435 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.503437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.504781 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.504793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.507555 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.507558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.507644 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.507666 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.507743 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.507823 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":59,"thread_start_us":60,"threads_started":1}
I20260924 10:47:01.508431 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.500087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.2.182.2:34641)
0924 10:47:01.500646 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.500660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.501290 (+ 630us) server_negotiation.cc:407] Connection header received
0924 10:47:01.501372 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.501378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.501573 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.501708 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.502031 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.503285 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.506581 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.507428 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.507540 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.507664 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.508191 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.508297 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":121,"threads_started":1}
I20260924 10:47:01.509259 16929 tablet_service.cc:1467] Processing CreateTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 (DEFAULT_TABLE table=test-table [id=8944bfbc60714838bc1053e73a642492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:01.509261 16661 tablet_service.cc:1467] Processing CreateTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 (DEFAULT_TABLE table=test-table [id=8944bfbc60714838bc1053e73a642492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:01.510219 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c383edf902a4cdfb00605a0ddef91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:01.510241 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c383edf902a4cdfb00605a0ddef91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:01.510851 16795 tablet_service.cc:1467] Processing CreateTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 (DEFAULT_TABLE table=test-table [id=8944bfbc60714838bc1053e73a642492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:01.511775 16795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c383edf902a4cdfb00605a0ddef91a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:01.516566 17012 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Bootstrap starting.
I20260924 10:47:01.518147 17014 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Bootstrap starting.
I20260924 10:47:01.518639 17013 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap starting.
I20260924 10:47:01.518689 17012 tablet_bootstrap.cc:654] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:01.519717 17012 log.cc:824] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:01.520603 17014 tablet_bootstrap.cc:654] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:01.520637 17013 tablet_bootstrap.cc:654] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:01.521432 17013 log.cc:824] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:01.521432 17014 log.cc:824] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:01.521600 17012 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: No bootstrap required, opened a new log
I20260924 10:47:01.521788 17012 ts_tablet_manager.cc:1397] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:47:01.523095 17013 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: No bootstrap required, opened a new log
I20260924 10:47:01.523329 17013 ts_tablet_manager.cc:1397] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:47:01.524953 17012 raft_consensus.cc:348] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.525197 17012 raft_consensus.cc:374] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:01.525256 17012 raft_consensus.cc:729] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70da2e0d20dc40d2860ebc03f4df0193, State: Initialized, Role: FOLLOWER
I20260924 10:47:01.525688 17012 consensus_queue.cc:260] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.526080 17014 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: No bootstrap required, opened a new log
I20260924 10:47:01.526253 17014 ts_tablet_manager.cc:1397] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 10:47:01.526363 16995 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
I20260924 10:47:01.526654 17012 ts_tablet_manager.cc:1428] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:47:01.527145 17013 raft_consensus.cc:348] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.527390 17013 raft_consensus.cc:374] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:01.527436 17013 raft_consensus.cc:729] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0847995a1b450b8e5e80d7961654a1, State: Initialized, Role: FOLLOWER
I20260924 10:47:01.527830 17013 consensus_queue.cc:260] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.528723 17013 ts_tablet_manager.cc:1428] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:47:01.529066 16727 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
I20260924 10:47:01.530697 17014 raft_consensus.cc:348] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.530956 17014 raft_consensus.cc:374] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:01.531030 17014 raft_consensus.cc:729] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fcc3192e92e481d910ca1420e571750, State: Initialized, Role: FOLLOWER
I20260924 10:47:01.531392 17014 consensus_queue.cc:260] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.531972 16861 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
I20260924 10:47:01.532259 17014 ts_tablet_manager.cc:1428] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 10:47:01.708832 16996 tablet.cc:2369] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:01.715512 17018 raft_consensus.cc:482] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:01.715683 17018 raft_consensus.cc:504] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.716791 17018 leader_election.cc:288] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641), ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423)
I20260924 10:47:01.720755 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.717115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.2.182.3:34687)
0924 10:47:01.717229 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.717245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.717347 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.717609 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.717612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.717632 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.717751 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.717756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.718678 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.718682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.719574 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.719583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.720316 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.720320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.720426 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.720458 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.720533 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.720607 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 10:47:01.720794 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.717191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:34687 (local address 127.2.182.1:36423)
0924 10:47:01.717328 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.717332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.717346 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:47:01.717444 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.717447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.717489 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.717584 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.717902 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.718574 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.719719 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.720206 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.720287 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.720339 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.720557 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.720630 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:47:01.721028 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.717120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:37443 (local address 127.2.182.2:34641)
0924 10:47:01.717255 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.717261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.717830 (+ 569us) server_negotiation.cc:407] Connection header received
0924 10:47:01.717929 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.717931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.717972 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.718032 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.718309 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.718888 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.719973 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.720426 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.720536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.720592 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.720787 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.720865 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:47:01.721383 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.717123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.2.182.3:37443)
0924 10:47:01.717731 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.717742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.717845 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.718061 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.718064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.718081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.718194 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.718201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.718992 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.718995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.719853 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.719863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.720562 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.720566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.720658 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.720681 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.720758 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.721254 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":291,"thread_start_us":132,"threads_started":1}
I20260924 10:47:01.721431 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "70da2e0d20dc40d2860ebc03f4df0193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0847995a1b450b8e5e80d7961654a1" is_pre_election: true
I20260924 10:47:01.721756 16681 raft_consensus.cc:2454] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70da2e0d20dc40d2860ebc03f4df0193 in term 0.
I20260924 10:47:01.721999 16815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "70da2e0d20dc40d2860ebc03f4df0193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcc3192e92e481d910ca1420e571750" is_pre_election: true
I20260924 10:47:01.722293 16815 raft_consensus.cc:2454] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70da2e0d20dc40d2860ebc03f4df0193 in term 0.
I20260924 10:47:01.722322 16884 leader_election.cc:302] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70da2e0d20dc40d2860ebc03f4df0193, ee0847995a1b450b8e5e80d7961654a1; no voters:
I20260924 10:47:01.722412 17020 raft_consensus.cc:482] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:01.722501 17020 raft_consensus.cc:504] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.722594 17018 raft_consensus.cc:2781] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:47:01.722716 17018 raft_consensus.cc:482] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:47:01.722770 17018 raft_consensus.cc:3037] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:01.723624 17020 leader_election.cc:288] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423), 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799)
I20260924 10:47:01.723961 17018 raft_consensus.cc:504] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.724336 17018 leader_election.cc:288] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [CANDIDATE]: Term 1 election: Requested vote from peers 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641), ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423)
I20260924 10:47:01.724804 16815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "70da2e0d20dc40d2860ebc03f4df0193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcc3192e92e481d910ca1420e571750"
I20260924 10:47:01.724942 16815 raft_consensus.cc:3037] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:01.725008 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "70da2e0d20dc40d2860ebc03f4df0193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0847995a1b450b8e5e80d7961654a1"
I20260924 10:47:01.725139 16681 raft_consensus.cc:3037] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:01.726192 16815 raft_consensus.cc:2454] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70da2e0d20dc40d2860ebc03f4df0193 in term 1.
I20260924 10:47:01.726436 16681 raft_consensus.cc:2454] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70da2e0d20dc40d2860ebc03f4df0193 in term 1.
I20260924 10:47:01.726497 16882 leader_election.cc:302] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193; no voters:
I20260924 10:47:01.726646 17018 raft_consensus.cc:2781] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:01.726850 17018 raft_consensus.cc:686] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 LEADER]: Becoming Leader. State: Replica: 70da2e0d20dc40d2860ebc03f4df0193, State: Running, Role: LEADER
I20260924 10:47:01.727501 17018 consensus_queue.cc:237] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:01.728227 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.723829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.2.182.2:58313)
0924 10:47:01.723930 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.723943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.724037 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.724474 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.724477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.724494 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.724604 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.724610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.725784 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.725788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.726992 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.727002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.727844 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.727924 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.727940 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.728024 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.728114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":37}
I20260924 10:47:01.728416 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.723979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38937 (local address 127.2.182.3:41799)
0924 10:47:01.724084 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.724090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.724445 (+ 355us) server_negotiation.cc:407] Connection header received
0924 10:47:01.724559 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.724564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.724609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.724675 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.725071 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.725727 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727232 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.727755 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727830 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.727879 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.728245 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.728318 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:47:01.728530 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.723835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:41799 (local address 127.2.182.2:38937)
0924 10:47:01.724365 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.724379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:01.724485 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:01.724744 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:01.724747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:01.724766 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:01.724936 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.724946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.725873 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.725876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.727094 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:01.727103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.727977 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:01.728084 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.728106 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:47:01.728215 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:47:01.728398 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":132,"thread_start_us":81,"threads_started":1}
I20260924 10:47:01.728544 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.724188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:58313 (local address 127.2.182.1:36423)
0924 10:47:01.724291 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.724295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.724308 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:47:01.724347 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.724349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.724397 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.724463 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:01.724909 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.725580 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727127 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.727637 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.727744 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.727797 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.728365 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.728432 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:47:01.728879 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "2fcc3192e92e481d910ca1420e571750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0847995a1b450b8e5e80d7961654a1" is_pre_election: true
I20260924 10:47:01.729089 16681 raft_consensus.cc:2379] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fcc3192e92e481d910ca1420e571750 in current term 1: Already voted for candidate 70da2e0d20dc40d2860ebc03f4df0193 in this term.
I20260924 10:47:01.729393 16949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c383edf902a4cdfb00605a0ddef91a9" candidate_uuid: "2fcc3192e92e481d910ca1420e571750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70da2e0d20dc40d2860ebc03f4df0193" is_pre_election: true
I20260924 10:47:01.730427 16749 leader_election.cc:302] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fcc3192e92e481d910ca1420e571750; no voters: 70da2e0d20dc40d2860ebc03f4df0193, ee0847995a1b450b8e5e80d7961654a1
I20260924 10:47:01.730939 17020 raft_consensus.cc:2726] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 10:47:01.731118 16536 catalog_manager.cc:5495] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "70da2e0d20dc40d2860ebc03f4df0193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } health_report { overall_health: HEALTHY } } }
W20260924 10:47:01.739341 16862 tablet.cc:2369] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:47:01.752295 16728 tablet.cc:2369] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:01.795109 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.791941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.2.182.3:41799)
0924 10:47:01.792081 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:01.792085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:01.792123 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:47:01.792195 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:01.792197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:01.792259 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:01.792338 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:01.792554 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.793075 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.793861 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:01.794063 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:01.794135 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:01.794389 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:01.794794 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:47:01.794896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:01.794947 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 10:47:01.794989 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:47:01.800796 16681 raft_consensus.cc:1261] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Refusing update from remote peer 70da2e0d20dc40d2860ebc03f4df0193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:01.800828 16815 raft_consensus.cc:1261] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Refusing update from remote peer 70da2e0d20dc40d2860ebc03f4df0193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:01.801472 17023 consensus_queue.cc:1035] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:01.801646 17018 consensus_queue.cc:1035] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:01.809145 17027 mvcc.cc:205] Tried to move back new op lower bound from 7332850982088794112 to 7332850981798948864. Current Snapshot: MvccSnapshot[applied={T|T < 7332850982088794112}]
I20260924 10:47:01.809165 17028 mvcc.cc:205] Tried to move back new op lower bound from 7332850982088794112 to 7332850981798948864. Current Snapshot: MvccSnapshot[applied={T|T < 7332850982088794112}]
I20260924 10:47:01.817384 16536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:01.817562 16536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:01.819937 16536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Sending DeleteTablet for 3 replicas of tablet 2c383edf902a4cdfb00605a0ddef91a9
I20260924 10:47:01.820708 16929 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:01 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260924 10:47:01.820708 16795 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:01 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:52782
I20260924 10:47:01.820748 16661 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:01 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36434
I20260924 10:47:01.821348 17036 tablet_replica.cc:330] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: stopping tablet replica
I20260924 10:47:01.821352 17037 tablet_replica.cc:330] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: stopping tablet replica
I20260924 10:47:01.821593 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16598
I20260924 10:47:01.821640 17037 raft_consensus.cc:2229] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:01.821640 17036 raft_consensus.cc:2229] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:47:01.821960 17037 raft_consensus.cc:2258] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:01.822188 17036 raft_consensus.cc:2258] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:01.823541 17037 ts_tablet_manager.cc:1905] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:01.824009 17036 ts_tablet_manager.cc:1905] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:01.826350 17037 ts_tablet_manager.cc:1918] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:01.826462 17037 log.cc:1196] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/2c383edf902a4cdfb00605a0ddef91a9
I20260924 10:47:01.826942 17036 ts_tablet_manager.cc:1918] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:01.827102 17036 log.cc:1196] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/2c383edf902a4cdfb00605a0ddef91a9
I20260924 10:47:01.827119 17037 ts_tablet_manager.cc:1939] T 2c383edf902a4cdfb00605a0ddef91a9 P 2fcc3192e92e481d910ca1420e571750: Deleting consensus metadata
I20260924 10:47:01.827497 17036 ts_tablet_manager.cc:1939] T 2c383edf902a4cdfb00605a0ddef91a9 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting consensus metadata
I20260924 10:47:01.828529 16521 catalog_manager.cc:4841] TS 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): tablet 2c383edf902a4cdfb00605a0ddef91a9 (table test-table [id=8944bfbc60714838bc1053e73a642492]) successfully deleted
I20260924 10:47:01.829012 16520 catalog_manager.cc:4841] TS 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): tablet 2c383edf902a4cdfb00605a0ddef91a9 (table test-table [id=8944bfbc60714838bc1053e73a642492]) successfully deleted
W20260924 10:47:01.830646 16522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:36423 (error 108)
I20260924 10:47:01.830922 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16732
I20260924 10:47:01.833178 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:01.832839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.0.0.1:36448)
0924 10:47:01.833015 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:01.833090 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 10:47:01.833360 16522 catalog_manager.cc:4584] TS ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c383edf902a4cdfb00605a0ddef91a9: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36423: connect: Connection refused (error 111)
I20260924 10:47:01.837332 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16866
I20260924 10:47:01.842335 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 16504
I20260924 10:47:01.847455 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--webserver_interface=127.2.182.62
--webserver_port=46167
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:01.954331 17040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:01.954632 17040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:01.954706 17040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:01.958650 17040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:47:01.958738 17040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:01.958776 17040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:01.958833 17040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:47:01.958870 17040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:47:01.964196 17040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:01.965234 17040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:01.966248 17040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:01.978972 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:01.978960 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:01.978947 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:01.979015 17040 server_base.cc:1034] running on GCE node
I20260924 10:47:01.979696 17040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:01.980768 17040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:01.981925 17040 hybrid_clock.cc:648] HybridClock initialized: now 1790246821981909 us; error 28 us; skew 500 ppm
Sep 24 10:47:01 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:01.984694 17040 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:47:01.985643 17040 webserver.cc:460] Webserver started at http://127.2.182.62:46167/ using document root <none> and password file <none>
I20260924 10:47:01.986200 17040 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:01.986304 17040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:01.989992 17040 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:47:01.991995 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:01.993261 17040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:01.993389 17040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "6cb719dd7b6a4b76b99e6018f62b1e18"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "52ba75d086600a94172e2bfbdbc429d8"
server_key_iv: "7879d9f4978df470f975cc8fa4018b85"
server_key_version: "encryptionkey@0"
I20260924 10:47:01.993957 17040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:02.021674 17040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:02.024523 17040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:02.024724 17040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:02.031613 17040 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32915
I20260924 10:47:02.031934 17108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32915 every 8 connection(s)
I20260924 10:47:02.032697 17040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:47:02.033851 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17040
I20260924 10:47:02.034366 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:36423
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=44083
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:02.037930 17109 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:47:02.039535 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap starting.
I20260924 10:47:02.056116 17109 log.cc:824] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:02.064845 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:02.065245 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap complete.
I20260924 10:47:02.069370 17109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:02.069787 17109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Initialized, Role: FOLLOWER
I20260924 10:47:02.070307 17109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:02.070449 17109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:47:02.070541 17109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:47:02.070654 17109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:47:02.072479 17109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:02.072816 17109 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb719dd7b6a4b76b99e6018f62b1e18; no voters:
I20260924 10:47:02.073171 17109 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:47:02.073258 17113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:47:02.073460 17113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 LEADER]: Becoming Leader. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Running, Role: LEADER
I20260924 10:47:02.073742 17113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:02.074370 17109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:47:02.074674 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb719dd7b6a4b76b99e6018f62b1e18. Latest consensus state: current_term: 2 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:02.074801 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:02.075268 17114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:02.075320 17117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:47:02.075377 17114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:02.080255 17117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:47:02.080876 17117 catalog_manager.cc:1259] Loaded cluster ID: fa4d3498e886402eb117dc559ca46e73
I20260924 10:47:02.080960 17117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:47:02.082355 17117 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 10:47:02.087291 17117 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 10:47:02.092463 17117 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:47:02.093225 17117 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Loaded TSK: 0
I20260924 10:47:02.094383 17117 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:47:02.146057 17111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:02.146353 17111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:02.146430 17111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:02.151005 17111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:02.151090 17111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:02.151193 17111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:47:02.156375 17111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:36423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=44083
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:02.157428 17111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:02.158414 17111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:02.168323 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:02.168218 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:02.168458 17111 server_base.cc:1034] running on GCE node
W20260924 10:47:02.168216 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:02.169059 17111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:02.169905 17111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:02.171105 17111 hybrid_clock.cc:648] HybridClock initialized: now 1790246822171079 us; error 39 us; skew 500 ppm
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:02.174364 17111 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:47:02.175354 17111 webserver.cc:460] Webserver started at http://127.2.182.1:44083/ using document root <none> and password file <none>
I20260924 10:47:02.176012 17111 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:02.176098 17111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:02.180270 17111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:47:02.182552 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.183805 17111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:47:02.183936 17111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "ee0847995a1b450b8e5e80d7961654a1"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "6745f1ae5b092164fe4ca249feae1784"
server_key_iv: "841488585b12a51d10f17c613958c105"
server_key_version: "encryptionkey@0"
I20260924 10:47:02.184469 17111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:02.240089 17111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:02.243613 17111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:02.243829 17111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:02.244446 17111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:02.245672 17153 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:47:02.249810 17111 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:47:02.249897 17111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 10:47:02.249997 17111 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:47:02.251649 17111 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:47:02.251720 17111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:47:02.251976 17153 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap starting.
I20260924 10:47:02.262217 17111 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36423
I20260924 10:47:02.262349 17260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:36423 every 8 connection(s)
I20260924 10:47:02.263185 17111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:47:02.270952 17153 log.cc:824] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:02.271727 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17111
I20260924 10:47:02.272140 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:34641
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=34983
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:02.280329 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.265216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:53501)
0924 10:47:02.265808 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.265860 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.266646 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.268042 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.268048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.268377 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:02.268861 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.268873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.270598 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.270600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.271529 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.271542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.272421 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.272427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.272676 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.273399 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:02.273424 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:02.274962 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:02.277219 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.277224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.277236 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.277489 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.277820 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.277825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.277826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.277959 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.278591 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:02.278596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:02.278845 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.279049 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.279457 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":136,"threads_started":1}
I20260924 10:47:02.280321 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.265282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:53501 (local address 127.2.182.62:32915)
0924 10:47:02.265675 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.265694 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.265998 (+ 304us) server_negotiation.cc:407] Connection header received
0924 10:47:02.267115 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.267134 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.267635 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.268006 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:02.269082 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.270458 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.271676 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.272250 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.272402 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.275161 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:02.275167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:02.275178 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:02.275203 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:02.277099 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.277607 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.277617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.277622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.277724 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.278058 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.278061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.278062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.278482 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:02.278619 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.279183 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.279450 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260924 10:47:02.281522 17153 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:02.282823 17153 tablet_bootstrap.cc:492] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap complete.
I20260924 10:47:02.282994 17261 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
I20260924 10:47:02.283215 17261 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:02.283695 17261 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:02.284287 17153 ts_tablet_manager.cc:1397] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.008s
I20260924 10:47:02.285939 17073 ts_manager.cc:194] Registered new tserver with Master: ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423)
I20260924 10:47:02.287076 17153 raft_consensus.cc:348] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:02.287532 17153 raft_consensus.cc:729] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0847995a1b450b8e5e80d7961654a1, State: Initialized, Role: FOLLOWER
I20260924 10:47:02.287973 17153 consensus_queue.cc:260] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } } peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } }
I20260924 10:47:02.288859 17153 ts_tablet_manager.cc:1428] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:47:02.290056 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:53501
I20260924 10:47:02.292251 17261 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
I20260924 10:47:02.295368 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.288746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.2.182.1:36423)
0924 10:47:02.291402 (+ 2656us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.291412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.291430 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:47:02.291484 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.291489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.291651 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.291786 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:02.292164 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.293125 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.294267 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.294733 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.294863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.295001 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.295165 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.295245 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2576,"thread_start_us":69,"threads_started":1}
I20260924 10:47:02.295368 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.288707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.0.0.1:36450)
0924 10:47:02.291009 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.291055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.291172 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.291916 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.291921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.291941 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:02.292043 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.292049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.293220 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.293225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.294159 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.294168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.294864 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.294868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.294976 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.295011 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.295139 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.295253 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"thread_start_us":92,"threads_started":1}
I20260924 10:47:02.296278 17195 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c383edf902a4cdfb00605a0ddef91a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:01 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36450
I20260924 10:47:02.297158 17270 tablet_replica.cc:330] stopping tablet replica
I20260924 10:47:02.297407 17270 raft_consensus.cc:2229] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:02.297654 17270 raft_consensus.cc:2258] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:02.298560 17270 ts_tablet_manager.cc:1905] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:02.301805 17270 ts_tablet_manager.cc:1918] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:02.301934 17270 log.cc:1196] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/2c383edf902a4cdfb00605a0ddef91a9
I20260924 10:47:02.302196 17270 ts_tablet_manager.cc:1939] T 2c383edf902a4cdfb00605a0ddef91a9 P ee0847995a1b450b8e5e80d7961654a1: Deleting consensus metadata
I20260924 10:47:02.303730 17059 catalog_manager.cc:4841] TS ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423): tablet 2c383edf902a4cdfb00605a0ddef91a9 (table test-table [id=8944bfbc60714838bc1053e73a642492]) successfully deleted
W20260924 10:47:02.383353 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:02.383587 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:02.383628 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:02.387473 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:02.387527 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:02.387583 17266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:47:02.393715 17266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:34641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=34983
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:02.394737 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:02.395691 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:02.405193 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:02.405233 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:02.405195 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:02.405469 17266 server_base.cc:1034] running on GCE node
I20260924 10:47:02.405849 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:02.406461 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:02.407652 17266 hybrid_clock.cc:648] HybridClock initialized: now 1790246822407626 us; error 44 us; skew 500 ppm
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:02.410328 17266 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:47:02.411137 17266 webserver.cc:460] Webserver started at http://127.2.182.2:34983/ using document root <none> and password file <none>
I20260924 10:47:02.411679 17266 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:02.411789 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:02.415397 17266 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:47:02.417351 17285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.418592 17266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:02.418731 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "2fcc3192e92e481d910ca1420e571750"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "22500390bab5493051e33e1632b9c1ca"
server_key_iv: "9a5aba3545654451129b7b207fcbbac5"
server_key_version: "encryptionkey@0"
I20260924 10:47:02.419238 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:02.435338 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:02.437887 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:02.438071 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:02.438606 17266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:02.439401 17266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:02.439469 17266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.439540 17266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:02.439610 17266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.448170 17266 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34641
I20260924 10:47:02.448192 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34641 every 8 connection(s)
I20260924 10:47:02.449167 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:02.459331 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17266
I20260924 10:47:02.459829 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:41799
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=40743
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:02.462194 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.450893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48743 (local address 127.2.182.62:32915)
0924 10:47:02.451055 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.451060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.452030 (+ 970us) server_negotiation.cc:407] Connection header received
0924 10:47:02.452777 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.452779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.452825 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.452909 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:02.454018 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.454527 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.455586 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.455814 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.455888 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.458360 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:02.458363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:02.458365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:02.458390 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:02.460035 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.460533 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.460539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.460544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.460637 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.460989 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.460995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.461000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.461264 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:02.461381 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.461970 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.462073 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:47:02.463126 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.451276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:48743)
0924 10:47:02.451889 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.451926 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.452607 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.453110 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.453117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.453446 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:02.453893 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.453906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.454624 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.454626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.455489 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.455497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.455932 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.455935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.456067 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.456603 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:02.456618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:02.458197 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:02.460139 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.460144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.460155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.460435 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.460726 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.460728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.460730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.460852 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.461384 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:02.461392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:02.461704 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.461918 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.462279 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":100,"threads_started":1}
I20260924 10:47:02.464275 17399 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
I20260924 10:47:02.464531 17399 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:02.465076 17399 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:02.466085 17073 ts_manager.cc:194] Registered new tserver with Master: 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
I20260924 10:47:02.466662 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:48743
W20260924 10:47:02.566824 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:02.567060 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:02.567124 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:02.570983 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:02.571059 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:02.571141 17402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:47:02.576474 17402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:41799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=40743
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:02.577497 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:02.578440 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:02.587781 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:02.587862 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:02.588069 17402 server_base.cc:1034] running on GCE node
W20260924 10:47:02.587862 17408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:02.588511 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:02.589170 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:02.590355 17402 hybrid_clock.cc:648] HybridClock initialized: now 1790246822590326 us; error 41 us; skew 500 ppm
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:02.592881 17402 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:47:02.593773 17402 webserver.cc:460] Webserver started at http://127.2.182.3:40743/ using document root <none> and password file <none>
I20260924 10:47:02.594247 17402 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:02.594313 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:02.598024 17402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:47:02.599923 17417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.601189 17402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:02.601320 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "70da2e0d20dc40d2860ebc03f4df0193"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "324ed140ffa05834e755059055590c59"
server_key_iv: "3c5f8389f783f0b1251526c0865fd3ff"
server_key_version: "encryptionkey@0"
I20260924 10:47:02.601846 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:02.623732 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:02.626838 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:02.627027 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:02.627601 17402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:02.628389 17402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:02.628434 17402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.628500 17402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:02.628537 17402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:02.637375 17402 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:41799
I20260924 10:47:02.637403 17530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:41799 every 8 connection(s)
I20260924 10:47:02.638340 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:47:02 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:02.646965 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17402
I20260924 10:47:02.651213 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.640012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:55523 (local address 127.2.182.62:32915)
0924 10:47:02.640171 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.640175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.641208 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 10:47:02.642034 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.642036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.642082 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.642155 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:02.643246 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.643763 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.644814 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.645043 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.645116 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.647379 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:02.647382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:02.647384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:02.647406 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:02.649191 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.649776 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.649778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.649779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.649851 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.650208 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.650212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.650213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.650418 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:02.650518 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.650950 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.651071 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:47:02.652117 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.640396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:55523)
0924 10:47:02.641077 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.641116 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.641863 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.642319 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.642325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.642661 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:02.643125 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.643138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.643864 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.643866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.644705 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.644723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.645157 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.645159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.645288 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.645776 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:02.645791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:02.647211 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:02.649311 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.649316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.649329 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.649677 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.649938 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:02.649941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:02.649943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:02.650115 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:02.650519 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:02.650523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:02.650727 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.650876 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.651143 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":101,"threads_started":1}
I20260924 10:47:02.653241 17531 heartbeater.cc:347] Connected to a master server at 127.2.182.62:32915
I20260924 10:47:02.653465 17531 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:02.653913 17531 heartbeater.cc:511] Master 127.2.182.62:32915 requested a full tablet report, sending...
I20260924 10:47:02.654994 17073 ts_manager.cc:194] Registered new tserver with Master: 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799)
I20260924 10:47:02.655673 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:55523
I20260924 10:47:02.656283 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.647602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.2.182.62:32915)
0924 10:47:02.647841 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.647847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.648093 (+ 246us) server_negotiation.cc:407] Connection header received
0924 10:47:02.648213 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.648215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.648281 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.648377 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:02.649311 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.650055 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.651110 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.651422 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.651528 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.652991 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:02.652994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:02.652995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:02.653015 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:02.654540 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.654991 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.654995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.654996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.655080 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:02.655398 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:02.655401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:02.655402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:02.655563 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:02.655698 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.655964 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.656082 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260924 10:47:02.657882 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:47:02.661981 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.658401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.2.182.62:32915)
0924 10:47:02.658536 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.658540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.658711 (+ 171us) server_negotiation.cc:407] Connection header received
0924 10:47:02.658880 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.658883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.658996 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.659097 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:02.659314 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.659803 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.660639 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.660877 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.661009 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.661301 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:02.661619 (+ 318us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:47:02.661742 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.661806 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.661859 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:47:02.662511 17073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:47:02.665747 17073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:47:02.673053 17195 tablet_service.cc:1467] Processing CreateTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 (DEFAULT_TABLE table=test-table [id=e3b81f8235eb4089bb71113e5fba408f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:02.673710 17195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d72bee674304fdbbedac3dadd7a5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:02.676501 17153 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap starting.
I20260924 10:47:02.677217 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.672072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:41799 (local address 127.0.0.1:37436)
0924 10:47:02.672190 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.672205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.672290 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.673196 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.673202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.673223 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:02.673340 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.673347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.674677 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.674681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.675811 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.675821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.676721 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.676725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.676814 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.676839 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.676998 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.677086 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:47:02.677286 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.672333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.2.182.3:41799)
0924 10:47:02.672674 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.672686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.672721 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:47:02.672791 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.672796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.673021 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.673218 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:02.673456 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.674532 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.675957 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.676605 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.676723 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.676880 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.677038 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.677137 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260924 10:47:02.677384 17153 tablet_bootstrap.cc:654] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:02.678292 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.672305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.0.0.1:52792)
0924 10:47:02.673135 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.673149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.673236 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.673935 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.673939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.673959 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:02.674082 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.674090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.675249 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.675253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.676527 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.676540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.677863 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.677866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.677956 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.677984 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.678053 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.678121 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":118,"threads_started":1}
I20260924 10:47:02.678812 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.672728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.2.182.2:34641)
0924 10:47:02.673099 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.673113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.673334 (+ 221us) server_negotiation.cc:407] Connection header received
0924 10:47:02.673417 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.673424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.673684 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.674008 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:02.674181 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.675099 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.676670 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.677528 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.677644 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.677794 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.678560 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.678658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20260924 10:47:02.679255 17153 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: No bootstrap required, opened a new log
I20260924 10:47:02.679354 17153 ts_tablet_manager.cc:1397] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:47:02.680025 17153 raft_consensus.cc:348] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.680119 17153 raft_consensus.cc:374] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:02.680145 17153 raft_consensus.cc:729] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0847995a1b450b8e5e80d7961654a1, State: Initialized, Role: FOLLOWER
I20260924 10:47:02.680276 17153 consensus_queue.cc:260] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.680536 17153 ts_tablet_manager.cc:1428] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:47:02.680567 17465 tablet_service.cc:1467] Processing CreateTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 (DEFAULT_TABLE table=test-table [id=e3b81f8235eb4089bb71113e5fba408f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:02.681643 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d72bee674304fdbbedac3dadd7a5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:02.681603 17333 tablet_service.cc:1467] Processing CreateTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 (DEFAULT_TABLE table=test-table [id=e3b81f8235eb4089bb71113e5fba408f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:02.682520 17333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d72bee674304fdbbedac3dadd7a5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:02.689916 17545 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Bootstrap starting.
I20260924 10:47:02.690418 17544 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Bootstrap starting.
I20260924 10:47:02.692373 17544 tablet_bootstrap.cc:654] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:02.692659 17545 tablet_bootstrap.cc:654] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:02.693619 17545 log.cc:824] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:02.693619 17544 log.cc:824] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:02.695601 17544 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: No bootstrap required, opened a new log
I20260924 10:47:02.695684 17545 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: No bootstrap required, opened a new log
I20260924 10:47:02.695791 17544 ts_tablet_manager.cc:1397] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:47:02.695840 17545 ts_tablet_manager.cc:1397] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:47:02.699730 17545 raft_consensus.cc:348] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.699724 17544 raft_consensus.cc:348] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.700027 17545 raft_consensus.cc:374] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:02.700027 17544 raft_consensus.cc:374] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:02.700080 17544 raft_consensus.cc:729] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70da2e0d20dc40d2860ebc03f4df0193, State: Initialized, Role: FOLLOWER
I20260924 10:47:02.700080 17545 raft_consensus.cc:729] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fcc3192e92e481d910ca1420e571750, State: Initialized, Role: FOLLOWER
I20260924 10:47:02.700558 17545 consensus_queue.cc:260] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.700558 17544 consensus_queue.cc:260] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.701254 17531 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
I20260924 10:47:02.701599 17545 ts_tablet_manager.cc:1428] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 10:47:02.701599 17544 ts_tablet_manager.cc:1428] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 10:47:02.702323 17399 heartbeater.cc:503] Master 127.2.182.62:32915 was elected leader, sending a full tablet report...
W20260924 10:47:02.764757 17262 tablet.cc:2369] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:02.815135 17267 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:02.815352 17267 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.816507 17267 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
I20260924 10:47:02.821700 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.816932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:45537 (local address 127.2.182.2:34641)
0924 10:47:02.817075 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.817080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.817098 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:47:02.817282 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.817285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.817338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.817426 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:02.817808 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.818589 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.820399 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.821020 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.821132 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.821193 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.821503 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.821570 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:47:02.821700 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.816802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:41513 (local address 127.2.182.3:41799)
0924 10:47:02.817076 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.817081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.817098 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:47:02.817282 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.817285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.817338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.817425 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:02.817810 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.818589 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.820399 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.821036 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.821132 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.821194 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.821503 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.821570 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260924 10:47:02.821760 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.816742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.2.182.1:45537)
0924 10:47:02.817025 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.817037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.817155 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.817452 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.817455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.817481 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:02.817684 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.817692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.818705 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.818708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.820271 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.820283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.821150 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.821154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.821310 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.821396 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.821475 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.821586 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":274,"spinlock_wait_cycles":27264,"thread_start_us":97,"threads_started":1}
I20260924 10:47:02.821995 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.816727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:41799 (local address 127.2.182.1:41513)
0924 10:47:02.817017 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.817033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:02.817155 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:02.817452 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:02.817455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:02.817481 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:02.817695 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.817701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.818705 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.818708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.820271 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:02.820283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.821150 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.821154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:02.821343 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.821401 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:47:02.821475 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:47:02.821586 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":355,"spinlock_wait_cycles":55040,"thread_start_us":71,"threads_started":1}
I20260924 10:47:02.822575 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d72bee674304fdbbedac3dadd7a5164" candidate_uuid: "ee0847995a1b450b8e5e80d7961654a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcc3192e92e481d910ca1420e571750" is_pre_election: true
I20260924 10:47:02.822618 17485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d72bee674304fdbbedac3dadd7a5164" candidate_uuid: "ee0847995a1b450b8e5e80d7961654a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70da2e0d20dc40d2860ebc03f4df0193" is_pre_election: true
I20260924 10:47:02.822888 17485 raft_consensus.cc:2454] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0847995a1b450b8e5e80d7961654a1 in term 0.
I20260924 10:47:02.822888 17353 raft_consensus.cc:2454] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0847995a1b450b8e5e80d7961654a1 in term 0.
I20260924 10:47:02.823412 17147 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcc3192e92e481d910ca1420e571750, ee0847995a1b450b8e5e80d7961654a1; no voters:
I20260924 10:47:02.823704 17267 raft_consensus.cc:2781] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:47:02.823791 17267 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:47:02.823835 17267 raft_consensus.cc:3037] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:02.824844 17267 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.825189 17267 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 1 election: Requested vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
I20260924 10:47:02.825489 17485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d72bee674304fdbbedac3dadd7a5164" candidate_uuid: "ee0847995a1b450b8e5e80d7961654a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70da2e0d20dc40d2860ebc03f4df0193"
I20260924 10:47:02.825544 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d72bee674304fdbbedac3dadd7a5164" candidate_uuid: "ee0847995a1b450b8e5e80d7961654a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcc3192e92e481d910ca1420e571750"
I20260924 10:47:02.825618 17485 raft_consensus.cc:3037] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:02.825675 17353 raft_consensus.cc:3037] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:02.826660 17353 raft_consensus.cc:2454] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0847995a1b450b8e5e80d7961654a1 in term 1.
I20260924 10:47:02.826666 17485 raft_consensus.cc:2454] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0847995a1b450b8e5e80d7961654a1 in term 1.
I20260924 10:47:02.826921 17147 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcc3192e92e481d910ca1420e571750, ee0847995a1b450b8e5e80d7961654a1; no voters:
I20260924 10:47:02.827095 17267 raft_consensus.cc:2781] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:02.827291 17267 raft_consensus.cc:686] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 LEADER]: Becoming Leader. State: Replica: ee0847995a1b450b8e5e80d7961654a1, State: Running, Role: LEADER
I20260924 10:47:02.827570 17267 consensus_queue.cc:237] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:02.830374 17072 catalog_manager.cc:5495] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1). New cstate: current_term: 1 leader_uuid: "ee0847995a1b450b8e5e80d7961654a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } health_report { overall_health: HEALTHY } } }
I20260924 10:47:02.867945 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.864155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.2.182.1:36423)
0924 10:47:02.864456 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:02.864462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:02.864473 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:47:02.864542 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:02.864544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:02.864595 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:02.864734 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:02.864994 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.865638 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.866467 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:02.866699 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:02.866818 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:02.867102 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:02.867521 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:47:02.867650 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:02.867712 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:47:02.867773 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260924 10:47:02.872102 17353 raft_consensus.cc:1261] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Refusing update from remote peer ee0847995a1b450b8e5e80d7961654a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:02.872102 17485 raft_consensus.cc:1261] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Refusing update from remote peer ee0847995a1b450b8e5e80d7961654a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:02.872859 17267 consensus_queue.cc:1035] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:02.873140 17552 consensus_queue.cc:1035] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:02.881569 17557 mvcc.cc:205] Tried to move back new op lower bound from 7332850986477637632 to 7332850986304204800. Current Snapshot: MvccSnapshot[applied={T|T < 7332850986477637632}]
I20260924 10:47:02.887910 17073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:02.888055 17073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:02.890269 17073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Sending DeleteTablet for 3 replicas of tablet 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:02.890910 17465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:02 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:37436
I20260924 10:47:02.890973 17333 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:02 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:52792
I20260924 10:47:02.891058 17195 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d72bee674304fdbbedac3dadd7a5164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:02 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36450
I20260924 10:47:02.891378 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17111
I20260924 10:47:02.891523 17566 tablet_replica.cc:330] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: stopping tablet replica
I20260924 10:47:02.892030 17567 tablet_replica.cc:330] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: stopping tablet replica
I20260924 10:47:02.892330 17567 raft_consensus.cc:2229] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:02.892630 17567 raft_consensus.cc:2258] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:02.891817 17566 raft_consensus.cc:2229] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:02.893469 17566 raft_consensus.cc:2258] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:02.894157 17567 ts_tablet_manager.cc:1905] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:02.894856 17566 ts_tablet_manager.cc:1905] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:02.896664 17567 ts_tablet_manager.cc:1918] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:02.896836 17567 log.cc:1196] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:02.897214 17567 ts_tablet_manager.cc:1939] T 7d72bee674304fdbbedac3dadd7a5164 P 70da2e0d20dc40d2860ebc03f4df0193: Deleting consensus metadata
I20260924 10:47:02.898365 17058 catalog_manager.cc:4841] TS 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): tablet 7d72bee674304fdbbedac3dadd7a5164 (table test-table [id=e3b81f8235eb4089bb71113e5fba408f]) successfully deleted
I20260924 10:47:02.898458 17566 ts_tablet_manager.cc:1918] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:02.898568 17566 log.cc:1196] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:02.898911 17566 ts_tablet_manager.cc:1939] T 7d72bee674304fdbbedac3dadd7a5164 P 2fcc3192e92e481d910ca1420e571750: Deleting consensus metadata
I20260924 10:47:02.899338 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17266
W20260924 10:47:02.901225 17059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:36423 (error 108)
I20260924 10:47:02.902144 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.901903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:36423 (local address 127.0.0.1:36454)
0924 10:47:02.902022 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.902076 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:47:02.902381 17059 catalog_manager.cc:4584] TS ee0847995a1b450b8e5e80d7961654a1 (127.2.182.1:36423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d72bee674304fdbbedac3dadd7a5164: Network error: Client connection negotiation failed: client connection to 127.2.182.1:36423: connect: Connection refused (error 111)
I20260924 10:47:02.905484 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17402
I20260924 10:47:02.906456 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:02.905612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.0.0.1:52796)
0924 10:47:02.906362 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:02.906402 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689}
I20260924 10:47:02.911083 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17040
I20260924 10:47:02.917151 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--webserver_interface=127.2.182.62
--webserver_port=46167
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:03.025820 17570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:03.026082 17570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:03.026152 17570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:03.030124 17570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:47:03.030187 17570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:03.030212 17570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:03.030229 17570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:47:03.030244 17570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:47:03.035472 17570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:03.036520 17570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:03.037573 17570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:03.045845 17578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.045830 17576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.045826 17575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:03.046161 17570 server_base.cc:1034] running on GCE node
I20260924 10:47:03.046684 17570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:03.047749 17570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:03.048954 17570 hybrid_clock.cc:648] HybridClock initialized: now 1790246823048932 us; error 25 us; skew 500 ppm
Sep 24 10:47:03 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:03.051671 17570 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:47:03.052654 17570 webserver.cc:460] Webserver started at http://127.2.182.62:46167/ using document root <none> and password file <none>
I20260924 10:47:03.053284 17570 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:03.053349 17570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:03.057283 17570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:47:03.059500 17585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.060814 17570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:03.060992 17570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "6cb719dd7b6a4b76b99e6018f62b1e18"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "52ba75d086600a94172e2bfbdbc429d8"
server_key_iv: "7879d9f4978df470f975cc8fa4018b85"
server_key_version: "encryptionkey@0"
I20260924 10:47:03.061565 17570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:03.078224 17570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:03.081333 17570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:03.081537 17570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:03.088609 17570 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:32915
I20260924 10:47:03.088879 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:32915 every 8 connection(s)
I20260924 10:47:03.089764 17570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:47:03.093185 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17570
I20260924 10:47:03.093629 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:36423
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=44083
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:03.093954 17638 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:47:03.095897 17638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap starting.
I20260924 10:47:03.120416 17638 log.cc:824] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:03.134212 17638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:03.134624 17638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18: Bootstrap complete.
I20260924 10:47:03.140129 17638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:03.140651 17638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Initialized, Role: FOLLOWER
I20260924 10:47:03.141217 17638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:03.141361 17638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:47:03.141427 17638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:47:03.141520 17638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:47:03.143957 17638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:03.144311 17638 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb719dd7b6a4b76b99e6018f62b1e18; no voters:
I20260924 10:47:03.144666 17638 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:47:03.144804 17642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:47:03.145110 17642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [term 3 LEADER]: Becoming Leader. State: Replica: 6cb719dd7b6a4b76b99e6018f62b1e18, State: Running, Role: LEADER
I20260924 10:47:03.145419 17642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } }
I20260924 10:47:03.145925 17638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:47:03.146283 17644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb719dd7b6a4b76b99e6018f62b1e18. Latest consensus state: current_term: 3 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:03.146412 17644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:03.146669 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb719dd7b6a4b76b99e6018f62b1e18" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 32915 } } }
I20260924 10:47:03.146747 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb719dd7b6a4b76b99e6018f62b1e18 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:03.146813 17646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:47:03.151785 17646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:47:03.152418 17646 catalog_manager.cc:1259] Loaded cluster ID: fa4d3498e886402eb117dc559ca46e73
I20260924 10:47:03.152480 17646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 10:47:03.153532 17646 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1414b159ad google::LogMessage::Fail() at ??:0
@ 0x7f1414b1888a google::LogMessage::SendToLog() at ??:0
@ 0x7f1414b154de google::LogMessage::Flush() at ??:0
@ 0x7f1414b16a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f141f454fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f141f455d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f141f452579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f141f496640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1416d673d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1416d67c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1416d693f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1416d58512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f141839c6db start_thread at ??:0
@ 0x7f141361361f clone at ??:0
W20260924 10:47:03.248571 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:03.248826 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:03.248873 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:03.253005 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:03.253059 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:03.253142 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:47:03.258394 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:36423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=44083
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:03.259418 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:03.260506 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:03.269763 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.269793 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.269831 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:03.270177 17640 server_base.cc:1034] running on GCE node
I20260924 10:47:03.270612 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:03.271368 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:03.272572 17640 hybrid_clock.cc:648] HybridClock initialized: now 1790246823272535 us; error 44 us; skew 500 ppm
Sep 24 10:47:03 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:03.275784 17640 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:47:03.276593 17640 webserver.cc:460] Webserver started at http://127.2.182.1:44083/ using document root <none> and password file <none>
I20260924 10:47:03.277210 17640 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:03.277300 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:03.281316 17640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:47:03.283453 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.284754 17640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:03.284845 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "ee0847995a1b450b8e5e80d7961654a1"
format_stamp: "Formatted at 2026-09-24 10:47:00 on dist-test-slave-6lwt"
server_key: "6745f1ae5b092164fe4ca249feae1784"
server_key_iv: "841488585b12a51d10f17c613958c105"
server_key_version: "encryptionkey@0"
I20260924 10:47:03.285430 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:03.305858 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:03.309177 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:03.309402 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:03.309984 17640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:03.311152 17696 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:47:03.316489 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:47:03.316565 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 10:47:03.316661 17640 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:47:03.318269 17640 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:47:03.318338 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:47:03.318686 17696 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap starting.
I20260924 10:47:03.328730 17640 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36423
I20260924 10:47:03.328992 17803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:36423 every 8 connection(s)
I20260924 10:47:03.329732 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:47:03.331157 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17640
I20260924 10:47:03.331637 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:34641
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=34983
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:03.340119 17696 log.cc:824] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:03.350872 17696 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:03.351270 17696 tablet_bootstrap.cc:492] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Bootstrap complete.
W20260924 10:47:03.351925 17806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:47:03.331755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:33431)
0924 10:47:03.332291 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.332348 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:03.333544 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:03.350988 (+ 17444us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":137,"threads_started":1}
W20260924 10:47:03.352545 17804 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:32915: Client connection negotiation failed: client connection to 127.2.182.62:32915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:47:03.352598 17696 ts_tablet_manager.cc:1397] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.010s
I20260924 10:47:03.353314 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.353049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:43951)
0924 10:47:03.353180 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.353249 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:47:03.353971 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.353745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:60187)
0924 10:47:03.353867 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.353915 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:47:03.354161 17804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:47:03.355329 17696 raft_consensus.cc:348] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:03.355753 17696 raft_consensus.cc:729] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0847995a1b450b8e5e80d7961654a1, State: Initialized, Role: FOLLOWER
I20260924 10:47:03.356189 17696 consensus_queue.cc:260] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:03.357247 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.357031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:56983)
0924 10:47:03.357141 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.357192 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:47:03.357254 17696 ts_tablet_manager.cc:1428] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260924 10:47:03.357900 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.357698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:38021)
0924 10:47:03.357800 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.357847 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:47:03.446354 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:03.446604 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:03.446646 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:03.450927 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:03.450991 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:03.451051 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:47:03.456535 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:34641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=34983
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:03.457656 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:03.458745 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:03.468621 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.468621 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.468617 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:03.468815 17807 server_base.cc:1034] running on GCE node
I20260924 10:47:03.469256 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:03.469882 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:03.471153 17807 hybrid_clock.cc:648] HybridClock initialized: now 1790246823471124 us; error 49 us; skew 500 ppm
Sep 24 10:47:03 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:03.473901 17807 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:47:03.474802 17807 webserver.cc:460] Webserver started at http://127.2.182.2:34983/ using document root <none> and password file <none>
I20260924 10:47:03.475366 17807 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:03.475458 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:03.479350 17807 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:47:03.481415 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.482612 17807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:03.482757 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "2fcc3192e92e481d910ca1420e571750"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "22500390bab5493051e33e1632b9c1ca"
server_key_iv: "9a5aba3545654451129b7b207fcbbac5"
server_key_version: "encryptionkey@0"
I20260924 10:47:03.483314 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:03.495059 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:03.497964 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:03.498163 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:03.498744 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:03.499557 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:03.499601 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.499672 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:03.499719 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.508941 17807 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:34641
I20260924 10:47:03.508993 17937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:34641 every 8 connection(s)
I20260924 10:47:03.509941 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:47:03.513351 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.511897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:58011)
0924 10:47:03.512521 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.512621 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":110,"threads_started":1}
W20260924 10:47:03.513756 17938 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:32915: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
I20260924 10:47:03.514281 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.514084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:47123)
0924 10:47:03.514180 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.514231 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 10:47:03.514793 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.514625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:60745)
0924 10:47:03.514711 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.514750 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 10:47:03.514961 17938 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:47:03.519229 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17807
I20260924 10:47:03.519606 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:41799
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:32915
--webserver_port=40743
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:03.625521 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:03.625860 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:03.625933 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:03.629917 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:03.629997 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:03.630084 17941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:47:03.635120 17941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:41799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=40743
--tserver_master_addrs=127.2.182.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:03.636097 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:03.637142 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:03.646281 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:03.646349 17941 server_base.cc:1034] running on GCE node
W20260924 10:47:03.646306 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:03.646289 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:03.646953 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:03.647490 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:03.648643 17941 hybrid_clock.cc:648] HybridClock initialized: now 1790246823648631 us; error 23 us; skew 500 ppm
Sep 24 10:47:03 dist-test-slave-6lwt krb5kdc[16491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:03.651207 17941 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:47:03.652006 17941 webserver.cc:460] Webserver started at http://127.2.182.3:40743/ using document root <none> and password file <none>
I20260924 10:47:03.652511 17941 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:03.652601 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:03.656116 17941 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:47:03.658004 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.659186 17941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:47:03.659300 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "70da2e0d20dc40d2860ebc03f4df0193"
format_stamp: "Formatted at 2026-09-24 10:47:01 on dist-test-slave-6lwt"
server_key: "324ed140ffa05834e755059055590c59"
server_key_iv: "3c5f8389f783f0b1251526c0865fd3ff"
server_key_version: "encryptionkey@0"
I20260924 10:47:03.659806 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:03.673029 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:03.675630 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:03.675815 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:03.676342 17941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:03.677151 17941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:03.677196 17941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.677260 17941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:03.677297 17941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:03.686116 17941 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:41799
I20260924 10:47:03.686200 18069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:41799 every 8 connection(s)
I20260924 10:47:03.687125 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:47:03.690739 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.689216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:44441)
0924 10:47:03.689950 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.690066 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":120,"threads_started":1}
W20260924 10:47:03.691147 18070 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:32915: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
I20260924 10:47:03.691704 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.691494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:54279)
0924 10:47:03.691600 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.691654 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:47:03.692271 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:03.692066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:55285)
0924 10:47:03.692164 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:03.692208 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:47:03.692447 18070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:47:03.695278 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 17941
I20260924 10:47:04.359054 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.358577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:50131)
0924 10:47:04.358892 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:04.358948 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260924 10:47:04.484299 18075 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:04.484479 18075 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:04.485644 18075 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
Sep 24 10:47:04 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.2@KRBTEST.COM
Sep 24 10:47:04 dist-test-slave-6lwt krb5kdc[16491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.3@KRBTEST.COM
I20260924 10:47:04.499578 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.485987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:46983 (local address 127.2.182.3:41799)
0924 10:47:04.486341 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:04.486357 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:04.486401 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:47:04.487120 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:04.487126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:04.487554 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:04.487813 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:04.489115 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.490164 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491362 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.491817 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491988 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:04.495137 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:04.495141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:04.495153 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:04.495179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:04.496772 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:04.497399 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:04.497405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:04.497411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:04.497571 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:04.497985 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:04.497990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:04.497992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:04.498440 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:04.498542 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:04.499127 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 10:47:04.499387 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260924 10:47:04.499594 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.486173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:33843 (local address 127.2.182.2:34641)
0924 10:47:04.486491 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:04.486501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:04.486532 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:47:04.487129 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:04.487134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:04.487545 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:04.487841 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:04.489124 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.490165 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491318 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.491809 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491963 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:04.495187 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:04.495191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:04.495202 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:04.495229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:04.497286 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:04.497784 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:04.497789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:04.497794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:04.497920 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:04.498190 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:04.498192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:04.498193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:04.498557 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:04.498679 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:04.499120 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 10:47:04.499403 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
I20260924 10:47:04.499606 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.485879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:34641 (local address 127.2.182.1:33843)
0924 10:47:04.486235 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:04.486250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:04.486353 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:04.488089 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:04.488092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:04.488449 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:04.488914 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:04.488951 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.490303 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.490310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:04.491174 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:04.491183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.491931 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:04.492161 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:04.492927 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:04.492949 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:04.494665 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:04.497399 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:04.497402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:04.497404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:04.497672 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:04.498013 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:04.498016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:04.498018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:04.498101 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:04.498664 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:04.498666 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:04.498798 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 10:47:04.499004 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 10:47:04.499361 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":73,"spinlock_wait_cycles":23680,"thread_start_us":85,"threads_started":1}
I20260924 10:47:04.499612 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.485802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:41799 (local address 127.2.182.1:46983)
0924 10:47:04.485942 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:04.485959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:04.486066 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:04.487893 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:04.487898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:04.488462 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:04.488951 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:04.488960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.490325 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.490328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:04.491181 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:04.491190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:04.491956 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:04.491960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:04.492183 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:04.492977 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:04.493013 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:04.494976 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:04.496915 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:04.496922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:04.496935 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:04.497272 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:04.497669 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:04.497671 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:04.497672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:04.497867 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:04.498557 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:04.498564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:04.498803 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 10:47:04.499003 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 10:47:04.499441 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":334,"spinlock_wait_cycles":23936}
W20260924 10:47:04.502147 17690 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
W20260924 10:47:04.502297 17691 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:04.502388 17690 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0847995a1b450b8e5e80d7961654a1; no voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193
I20260924 10:47:04.502664 18075 raft_consensus.cc:2726] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:47:04.515722 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.515394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:42707)
0924 10:47:04.515591 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:04.515637 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260924 10:47:04.693325 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:04.692966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:44003)
0924 10:47:04.693191 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:04.693239 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260924 10:47:05.360303 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:05.359903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:53669)
0924 10:47:05.360154 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:05.360207 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260924 10:47:05.516777 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:05.516371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:57403)
0924 10:47:05.516620 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:05.516676 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260924 10:47:05.694406 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:05.694000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:47455)
0924 10:47:05.694245 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:05.694301 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260924 10:47:06.361611 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:06.361128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:54307)
0924 10:47:06.361453 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:06.361511 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":93,"threads_started":1}
I20260924 10:47:06.517879 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:06.517469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:34405)
0924 10:47:06.517739 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:06.517789 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260924 10:47:06.695415 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:06.695038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:53445)
0924 10:47:06.695284 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:06.695333 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260924 10:47:06.752337 18087 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:06.752486 18087 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:06.752985 18087 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
W20260924 10:47:06.753726 17690 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
W20260924 10:47:06.753830 17691 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:06.753945 17691 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0847995a1b450b8e5e80d7961654a1; no voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193
I20260924 10:47:06.754098 18087 raft_consensus.cc:2726] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:47:07.363051 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:07.362383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:39827)
0924 10:47:07.362686 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:07.362880 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260924 10:47:07.519196 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:07.518781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:35643)
0924 10:47:07.519054 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:07.519106 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260924 10:47:07.696591 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:07.696179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:45029)
0924 10:47:07.696456 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:07.696509 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260924 10:47:08.364750 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:08.364208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:51241)
0924 10:47:08.364540 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:08.364636 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260924 10:47:08.520380 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:08.519945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:39679)
0924 10:47:08.520233 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:08.520285 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":132,"threads_started":1}
I20260924 10:47:08.551980 18093 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:08.552151 18093 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:08.552632 18093 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
W20260924 10:47:08.553506 17690 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
W20260924 10:47:08.553689 17691 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:08.553781 17691 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0847995a1b450b8e5e80d7961654a1; no voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193
I20260924 10:47:08.553964 18093 raft_consensus.cc:2726] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:47:08.697902 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:08.697441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:52747)
0924 10:47:08.697722 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:08.697771 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260924 10:47:09.366083 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:09.365649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:33483)
0924 10:47:09.365921 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:09.365976 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260924 10:47:09.521613 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:09.521170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:59951)
0924 10:47:09.521448 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:09.521506 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
I20260924 10:47:09.699069 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:09.698668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:41619)
0924 10:47:09.698924 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:09.698977 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260924 10:47:10.367244 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:10.366864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:49975)
0924 10:47:10.367100 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:10.367153 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260924 10:47:10.522810 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:10.522335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:35361)
0924 10:47:10.522659 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:10.522714 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260924 10:47:10.700205 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:10.699763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:48479)
0924 10:47:10.700041 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:10.700093 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":65,"threads_started":1}
I20260924 10:47:11.368517 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:11.368121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:36749)
0924 10:47:11.368370 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:11.368423 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260924 10:47:11.524068 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:11.523654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:54887)
0924 10:47:11.523921 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:11.523975 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260924 10:47:11.701309 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:11.700872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:43709)
0924 10:47:11.701160 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:11.701217 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260924 10:47:12.369787 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:12.369356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:42009)
0924 10:47:12.369624 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:12.369684 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260924 10:47:12.525246 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:12.524762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:60143)
0924 10:47:12.525085 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:12.525148 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":77,"threads_started":1}
I20260924 10:47:12.702617 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:12.702078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:56531)
0924 10:47:12.702416 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:12.702479 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260924 10:47:13.371151 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:13.370649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:53851)
0924 10:47:13.370927 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:13.371020 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260924 10:47:13.492363 18108 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:13.492493 18108 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:13.493001 18108 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
W20260924 10:47:13.493748 17690 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
W20260924 10:47:13.493835 17691 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:13.493924 17691 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0847995a1b450b8e5e80d7961654a1; no voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193
I20260924 10:47:13.494072 18108 raft_consensus.cc:2726] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:47:13.526335 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:13.525952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:41211)
0924 10:47:13.526193 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:13.526242 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260924 10:47:13.703835 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:13.703375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:49885)
0924 10:47:13.703674 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:13.703738 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260924 10:47:14.372336 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:14.371949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:51341)
0924 10:47:14.372188 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:14.372243 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260924 10:47:14.527400 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:14.526999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:45391)
0924 10:47:14.527260 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:14.527313 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260924 10:47:14.704970 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:14.704563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:37045)
0924 10:47:14.704798 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:14.704850 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260924 10:47:15.373636 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:15.373177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:36403)
0924 10:47:15.373455 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:15.373510 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260924 10:47:15.528481 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:15.528099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:34737)
0924 10:47:15.528337 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:15.528390 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260924 10:47:15.641119 18116 raft_consensus.cc:482] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:15.641243 18116 raft_consensus.cc:504] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70da2e0d20dc40d2860ebc03f4df0193" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 41799 } } peers { permanent_uuid: "2fcc3192e92e481d910ca1420e571750" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 34641 } } peers { permanent_uuid: "ee0847995a1b450b8e5e80d7961654a1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36423 } }
I20260924 10:47:15.641664 18116 leader_election.cc:288] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799), 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641)
W20260924 10:47:15.642469 17691 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70da2e0d20dc40d2860ebc03f4df0193 (127.2.182.3:41799): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
W20260924 10:47:15.642560 17690 leader_election.cc:341] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fcc3192e92e481d910ca1420e571750 (127.2.182.2:34641): Not found: Tablet not found: 7d72bee674304fdbbedac3dadd7a5164
I20260924 10:47:15.642630 17690 leader_election.cc:302] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0847995a1b450b8e5e80d7961654a1; no voters: 2fcc3192e92e481d910ca1420e571750, 70da2e0d20dc40d2860ebc03f4df0193
I20260924 10:47:15.642782 18116 raft_consensus.cc:2726] T 7d72bee674304fdbbedac3dadd7a5164 P ee0847995a1b450b8e5e80d7961654a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:47:15.706096 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:15.705697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:34183)
0924 10:47:15.705954 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:15.706007 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260924 10:47:16.374909 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:16.374505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:59933)
0924 10:47:16.374766 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:16.374815 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260924 10:47:16.529707 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:16.529279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:44973)
0924 10:47:16.529557 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:16.529613 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260924 10:47:16.707176 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:16.706791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:45907)
0924 10:47:16.707016 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:16.707083 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260924 10:47:17.376181 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:17.375740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:38089)
0924 10:47:17.376017 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:17.376079 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260924 10:47:17.530902 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:17.530460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:37641)
0924 10:47:17.530748 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:17.530801 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260924 10:47:17.708381 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:17.707982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:44287)
0924 10:47:17.708211 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:17.708265 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260924 10:47:18.377779 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:18.377332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.1:39933)
0924 10:47:18.377606 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:18.377669 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
I20260924 10:47:18.532383 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:18.531899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.2:53225)
0924 10:47:18.532217 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:18.532289 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260924 10:47:18.708243 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17640
I20260924 10:47:18.710032 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:18.709541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:32915 (local address 127.2.182.3:39113)
0924 10:47:18.709875 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:18.709945 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":67,"threads_started":1}
I20260924 10:47:18.714318 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17807
I20260924 10:47:18.719856 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 17941
2026-09-24T10:47:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19554 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:47:19 dist-test-slave-6lwt krb5kdc[18129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:47:19 dist-test-slave-6lwt krb5kdc[18129](info): set up 2 sockets
Sep 24 10:47:19 dist-test-slave-6lwt krb5kdc[18129](info): commencing operation
krb5kdc: starting...
W20260924 10:47:20.009275 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:47:20.130206 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:20.238660 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:20.238947 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:20.239014 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:20.243032 18142 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:47:20.243108 18142 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:47:20.243135 18142 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:47:20.243171 18142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:47:20.243212 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:20.243266 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:20.243304 18142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:47:20.243340 18142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:47:20.248413 18142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:20.249478 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:20.250447 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:20.258663 18148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.258674 18147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.258674 18150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.258765 18142 server_base.cc:1034] running on GCE node
I20260924 10:47:20.259419 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:20.260447 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:20.261636 18142 hybrid_clock.cc:648] HybridClock initialized: now 1790246840261622 us; error 26 us; skew 500 ppm
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:20.264160 18142 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:47:20.265056 18142 webserver.cc:460] Webserver started at http://127.2.182.62:45133/ using document root <none> and password file <none>
I20260924 10:47:20.265600 18142 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:20.265681 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:20.265904 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:20.267840 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "07be7f40dc2342c7e2dc84b3d038ff23"
server_key_iv: "a297f6bc07daa7c881c38885b238aed2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.268345 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "07be7f40dc2342c7e2dc84b3d038ff23"
server_key_iv: "a297f6bc07daa7c881c38885b238aed2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.271744 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 10:47:20.273953 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.275046 18142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:47:20.275172 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "07be7f40dc2342c7e2dc84b3d038ff23"
server_key_iv: "a297f6bc07daa7c881c38885b238aed2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.275269 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:20.290535 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:20.293282 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:20.293480 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:20.300071 18142 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:44635
I20260924 10:47:20.300354 18209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:44635 every 8 connection(s)
I20260924 10:47:20.301009 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:47:20.303767 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:20.305905 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18142
I20260924 10:47:20.306046 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:47:20.306348 2776 external_mini_cluster.cc:1428] Setting key 2d94556af60968edc8f6ae99fa12d509
I20260924 10:47:20.310212 18210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap starting.
I20260924 10:47:20.312741 18210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:20.313458 18210 log.cc:824] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:20.315366 18210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: No bootstrap required, opened a new log
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:20.318430 18210 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:20.318655 18210 raft_consensus.cc:374] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:20.318727 18210 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Initialized, Role: FOLLOWER
I20260924 10:47:20.319187 18210 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:20.319339 18210 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:47:20.319401 18210 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:47:20.319492 18210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:20.320565 18210 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:20.320935 18210 leader_election.cc:302] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96ed11b1cd6408ea2d6d0a71ac5ed25; no voters:
I20260924 10:47:20.321242 18210 leader_election.cc:288] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:47:20.321288 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:20.307700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.2.182.62:44635)
0924 10:47:20.308106 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:20.308127 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:20.308171 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:47:20.309130 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:20.309152 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:20.309546 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:20.309896 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:20.310598 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.312044 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.312966 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.313414 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.313578 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:20.316052 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:20.316060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:20.316082 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:20.316124 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:20.318157 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.318591 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.318596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.318600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.318767 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.319049 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.319053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.319054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.319558 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:20.319776 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:20.320014 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 10:47:20.320325 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20260924 10:47:20.321368 18215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:20.321571 18215 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 LEADER]: Becoming Leader. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Running, Role: LEADER
I20260924 10:47:20.321888 18215 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:20.322400 18210 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:47:20.323462 18216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:47:20.323652 18216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:20.323936 18216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96ed11b1cd6408ea2d6d0a71ac5ed25. Latest consensus state: current_term: 1 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:47:20.324088 18216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
W20260924 10:47:20.326783 18230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:47:20.326869 18230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:47:20.326943 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:47:20.327648 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:47:20.332598 18224 catalog_manager.cc:1347] Generated new cluster ID: ecb78fcb5c9547a4a6315377180ec554
I20260924 10:47:20.332655 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:47:20.368494 18224 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:47:20.369508 18224 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:47:20.380116 18224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Generated new TSK 0
I20260924 10:47:20.380813 18224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:47:20.411926 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:20.548064 18236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:20.548332 18236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:20.548375 18236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:20.552294 18236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:20.552348 18236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:20.552407 18236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:47:20.557478 18236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:20.558507 18236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:20.559571 18236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:20.568847 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.568821 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.569012 18236 server_base.cc:1034] running on GCE node
W20260924 10:47:20.568949 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.569541 18236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:20.570159 18236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:20.571336 18236 hybrid_clock.cc:648] HybridClock initialized: now 1790246840571321 us; error 30 us; skew 500 ppm
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:20.573976 18236 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:47:20.574782 18236 webserver.cc:460] Webserver started at http://127.2.182.1:36827/ using document root <none> and password file <none>
I20260924 10:47:20.575323 18236 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:20.575408 18236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:20.575651 18236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:20.577656 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "f75e33712d5f412daf6a697d50c98a49"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1dcb3cbe5bda0ddd0771363cd7b5686"
server_key_iv: "eb34d7be628c2e8b18f90547e400a1e2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.578253 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "f75e33712d5f412daf6a697d50c98a49"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1dcb3cbe5bda0ddd0771363cd7b5686"
server_key_iv: "eb34d7be628c2e8b18f90547e400a1e2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.581789 18236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:47:20.583978 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.585156 18236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:20.585271 18236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "f75e33712d5f412daf6a697d50c98a49"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1dcb3cbe5bda0ddd0771363cd7b5686"
server_key_iv: "eb34d7be628c2e8b18f90547e400a1e2"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.585369 18236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:20.608568 18236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:20.611395 18236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:20.611595 18236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:20.612190 18236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:20.613127 18236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:20.613183 18236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.613230 18236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:20.613256 18236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.621866 18236 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:45697
I20260924 10:47:20.621891 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:45697 every 8 connection(s)
I20260924 10:47:20.622769 18236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:47:20.628795 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18236
I20260924 10:47:20.628952 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:47:20.629225 2776 external_mini_cluster.cc:1428] Setting key 8bf699e1cf978af7fa5d3949e7517cac
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:20.636384 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:20.624505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:56411 (local address 127.2.182.62:44635)
0924 10:47:20.624644 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:20.624648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:20.625447 (+ 799us) server_negotiation.cc:407] Connection header received
0924 10:47:20.626742 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:20.626745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:20.626792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:20.626865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:20.627925 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.628442 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.629673 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.629898 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.630000 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:20.632524 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:20.632529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:20.632534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:20.632565 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:20.634242 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.634739 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.634746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.634749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.634865 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.635186 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.635189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.635191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.635444 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:20.635560 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:20.635983 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:47:20.636137 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":48}
I20260924 10:47:20.636965 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:20.624842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:56411)
0924 10:47:20.625306 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:20.625345 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:20.626555 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:20.627010 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:20.627016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:20.627343 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:20.627806 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:20.627819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.628538 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.628540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:20.629547 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:20.629556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.630031 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.630035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:20.630210 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:20.630738 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:20.630753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:20.632371 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:20.634357 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:20.634362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:20.634372 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:20.634628 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:20.634958 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:20.634961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:20.634962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:20.635097 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:20.635543 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:20.635547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:20.635756 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 10:47:20.635900 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:47:20.636139 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260924 10:47:20.638167 18365 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:20.638381 18365 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:47:20.638803 18365 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:20.640210 18174 ts_manager.cc:194] Registered new tserver with Master: f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:47:20.641463 18174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:56411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:47:20.655752 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:20.761162 18370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:20.761409 18370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:20.761448 18370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:20.765413 18370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:20.765465 18370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:20.765563 18370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:47:20.770730 18370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:20.771708 18370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:20.772686 18370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:20.782100 18378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.782200 18375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.782153 18370 server_base.cc:1034] running on GCE node
W20260924 10:47:20.782107 18376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.782724 18370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:20.783332 18370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:20.784502 18370 hybrid_clock.cc:648] HybridClock initialized: now 1790246840784512 us; error 71 us; skew 500 ppm
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:20.787117 18370 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:47:20.787901 18370 webserver.cc:460] Webserver started at http://127.2.182.2:37397/ using document root <none> and password file <none>
I20260924 10:47:20.788394 18370 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:20.788450 18370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:20.788600 18370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:20.790565 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "63b28dc8c6c048c1b66276fd5775f07b"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1d090fda72fe7bea9b34689b5016e9f"
server_key_iv: "53d6deb09b288b068ea8029531464de1"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.791105 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "63b28dc8c6c048c1b66276fd5775f07b"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1d090fda72fe7bea9b34689b5016e9f"
server_key_iv: "53d6deb09b288b068ea8029531464de1"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.794704 18370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:47:20.796983 18385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.798197 18370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:20.798285 18370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "63b28dc8c6c048c1b66276fd5775f07b"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1d090fda72fe7bea9b34689b5016e9f"
server_key_iv: "53d6deb09b288b068ea8029531464de1"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.798401 18370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:20.813516 18370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:20.816253 18370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:20.816434 18370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:20.817014 18370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:20.817759 18370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:20.817802 18370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.817867 18370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:20.817907 18370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:20.826720 18370 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:32997
I20260924 10:47:20.826761 18498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:32997 every 8 connection(s)
I20260924 10:47:20.827561 18370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:47:20.831255 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18370
I20260924 10:47:20.831393 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:47:20.831723 2776 external_mini_cluster.cc:1428] Setting key 8bfabad78d05cd9483996ca39f2b44b5
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:47:20.841804 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:20.829190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:56949 (local address 127.2.182.62:44635)
0924 10:47:20.829314 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:20.829317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:20.830541 (+ 1224us) server_negotiation.cc:407] Connection header received
0924 10:47:20.831403 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:20.831407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:20.831465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:20.831563 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:20.832970 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.833755 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.834917 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.835142 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.835232 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:20.838089 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:20.838099 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:20.838101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:20.838126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:20.839643 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.840173 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.840179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.840181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.840269 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:20.840626 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:20.840632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:20.840634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:20.840836 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:20.840984 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:20.841484 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 10:47:20.841616 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:47:20.842641 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:20.829575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:56949)
0924 10:47:20.830394 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:20.830432 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:20.831178 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:20.831703 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:20.831709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:20.832143 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:20.832754 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:20.832773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.833887 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.833890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:20.834772 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:20.834785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:20.835287 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:20.835293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:20.835540 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:20.836164 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:20.836189 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:20.837920 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:20.839770 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:20.839777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:20.839791 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:20.840066 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:20.840369 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:20.840374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:20.840376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:20.840524 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:20.840979 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:20.840985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:20.841235 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 10:47:20.841421 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:47:20.841742 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":531,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
I20260924 10:47:20.843880 18499 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:20.844105 18499 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:20.844537 18499 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:20.845698 18174 ts_manager.cc:194] Registered new tserver with Master: 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
I20260924 10:47:20.846587 18174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:56949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:47:20.859532 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:20.966840 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:20.967087 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:20.967164 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:20.971153 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:20.971207 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:20.971261 18504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:47:20.976378 18504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:20.977438 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:20.978572 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:20.987994 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.987962 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:20.987979 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:20.988246 18504 server_base.cc:1034] running on GCE node
I20260924 10:47:20.988801 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:20.989478 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:20.990674 18504 hybrid_clock.cc:648] HybridClock initialized: now 1790246840990650 us; error 40 us; skew 500 ppm
Sep 24 10:47:20 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:20.993359 18504 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:47:20.994186 18504 webserver.cc:460] Webserver started at http://127.2.182.3:43317/ using document root <none> and password file <none>
I20260924 10:47:20.994778 18504 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:20.994871 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:20.995087 18504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:47:20.997100 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "a4608a43dd814c138371a63fa03079d7"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a4f762acbe41c80cf306d3b3beffdaa9"
server_key_iv: "51ac588156fc69dc20ac90a482398ed1"
server_key_version: "encryptionkey@0"
I20260924 10:47:20.997627 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "a4608a43dd814c138371a63fa03079d7"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a4f762acbe41c80cf306d3b3beffdaa9"
server_key_iv: "51ac588156fc69dc20ac90a482398ed1"
server_key_version: "encryptionkey@0"
I20260924 10:47:21.001277 18504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:47:21.004012 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:21.005293 18504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:21.005411 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "a4608a43dd814c138371a63fa03079d7"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a4f762acbe41c80cf306d3b3beffdaa9"
server_key_iv: "51ac588156fc69dc20ac90a482398ed1"
server_key_version: "encryptionkey@0"
I20260924 10:47:21.005513 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:21.030263 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:21.033022 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:21.033215 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:21.033789 18504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:21.034582 18504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:21.034627 18504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:21.034688 18504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:21.034725 18504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:21.042896 18504 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44073
I20260924 10:47:21.042930 18632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44073 every 8 connection(s)
I20260924 10:47:21.043744 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:47:21.045351 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18504
I20260924 10:47:21.045483 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:47:21.045746 2776 external_mini_cluster.cc:1428] Setting key 8edd4886946be226d92cf99994d5f083
Sep 24 10:47:21 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:21.056986 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.045395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:36797 (local address 127.2.182.62:44635)
0924 10:47:21.045522 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.045526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.046475 (+ 949us) server_negotiation.cc:407] Connection header received
0924 10:47:21.048035 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.048038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.048101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.048194 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:21.049372 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.049903 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.051019 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.051247 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.051355 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.053737 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:21.053745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:21.053748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:21.053772 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:21.055014 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:21.055497 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:21.055501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:21.055503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:21.055583 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:21.055919 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:21.055924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:21.055926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:21.056111 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:21.056216 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.056673 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.056793 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:47:21.057730 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.045812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:36797)
0924 10:47:21.046320 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.046373 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.047848 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.048363 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.048369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.048710 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:21.049240 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.049253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.050020 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.050024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.050898 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.050907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.051385 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.051389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.051558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.052106 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:21.052122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:21.053565 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:21.055134 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:21.055140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:21.055154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:21.055395 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:21.055697 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:21.055705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:21.055706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:21.055826 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:21.056213 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:21.056219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:21.056444 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.056635 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.056965 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":144,"threads_started":1}
I20260924 10:47:21.058904 18633 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:21.059134 18633 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:21.059630 18633 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:21.060549 18174 ts_manager.cc:194] Registered new tserver with Master: a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073)
I20260924 10:47:21.061115 18174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:36797
I20260924 10:47:21.070876 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:47:21.079804 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.072730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.2.182.62:44635)
0924 10:47:21.072853 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.072858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.073062 (+ 204us) server_negotiation.cc:407] Connection header received
0924 10:47:21.073200 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.073203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.073246 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.073323 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:21.073982 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.074466 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.075278 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.075484 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.075584 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.076842 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:21.076846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:21.076849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:21.076877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:21.078119 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:21.078560 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:21.078565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:21.078567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:21.078647 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:21.078915 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:21.078918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:21.078920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:21.079132 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:21.079267 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.079513 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.079622 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:47:21.082432 18174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:47:21.085460 18174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:47:21.100255 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.093460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.0.0.1:55150)
0924 10:47:21.094055 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.094103 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.094218 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.095298 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.095304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.095326 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:21.095483 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.095491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.096984 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.096987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.098351 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.098360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.099478 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.099483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.099741 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.099768 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.099883 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.100078 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":158,"thread_start_us":92,"threads_started":1}
I20260924 10:47:21.100260 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.093637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.0.0.1:53280)
0924 10:47:21.094318 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.094332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.094414 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.095335 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.095337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.095359 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:21.095515 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.095522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.096715 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.096718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.098345 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.098351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.099478 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.099483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.099721 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.099768 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.099883 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.100078 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":217,"thread_start_us":114,"threads_started":1}
I20260924 10:47:21.100730 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.094632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.2.182.2:32997)
0924 10:47:21.094954 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.094964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.094981 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:47:21.095040 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.095044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.095190 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.095678 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:21.095826 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.096555 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.098454 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.099327 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.099440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.099584 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.100326 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.100543 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
I20260924 10:47:21.100980 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44073 (local address 127.0.0.1:50636)
0924 10:47:21.094448 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.094460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.094541 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.095512 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.095515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.095530 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:21.095632 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.095638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.097219 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.097225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.098334 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.098345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.100583 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.100586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.100678 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.100695 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.100735 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.100805 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":105,"thread_start_us":85,"threads_started":1}
I20260924 10:47:21.101210 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.093596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.2.182.1:45697)
0924 10:47:21.094796 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.094815 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.094840 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:47:21.094917 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.094922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.095136 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.095305 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:21.095627 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.096851 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.098494 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.099327 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.099440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.099583 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.100968 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.101083 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":90,"threads_started":1}
I20260924 10:47:21.101706 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.094709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.2.182.3:44073)
0924 10:47:21.095065 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.095074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.095092 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:47:21.095151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.095154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.095332 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.095520 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:21.095720 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.097076 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.099646 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.100435 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.101277 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.101434 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.101482 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.101578 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260924 10:47:21.103768 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3299587b24102b4d1ca41dce426f2 (DEFAULT_TABLE table=test-table [id=ed91b3dad91e452d840988981db7e7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:21.103771 18433 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3299587b24102b4d1ca41dce426f2 (DEFAULT_TABLE table=test-table [id=ed91b3dad91e452d840988981db7e7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:21.104257 18567 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3299587b24102b4d1ca41dce426f2 (DEFAULT_TABLE table=test-table [id=ed91b3dad91e452d840988981db7e7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:21.104990 18433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3299587b24102b4d1ca41dce426f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:21.104990 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3299587b24102b4d1ca41dce426f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:21.105279 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3299587b24102b4d1ca41dce426f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:21.111207 18650 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Bootstrap starting.
I20260924 10:47:21.112613 18651 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Bootstrap starting.
I20260924 10:47:21.112989 18652 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Bootstrap starting.
I20260924 10:47:21.113772 18650 tablet_bootstrap.cc:654] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:21.114665 18651 tablet_bootstrap.cc:654] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:21.114784 18650 log.cc:824] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:21.115168 18652 tablet_bootstrap.cc:654] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:21.115829 18651 log.cc:824] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:21.115829 18652 log.cc:824] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:21.116623 18650 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: No bootstrap required, opened a new log
I20260924 10:47:21.116813 18650 ts_tablet_manager.cc:1397] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:47:21.117836 18651 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: No bootstrap required, opened a new log
I20260924 10:47:21.118024 18651 ts_tablet_manager.cc:1397] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:47:21.118245 18652 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: No bootstrap required, opened a new log
I20260924 10:47:21.118446 18652 ts_tablet_manager.cc:1397] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:47:21.119498 18650 raft_consensus.cc:348] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.119725 18650 raft_consensus.cc:374] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:21.119774 18650 raft_consensus.cc:729] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4608a43dd814c138371a63fa03079d7, State: Initialized, Role: FOLLOWER
I20260924 10:47:21.120190 18650 consensus_queue.cc:260] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.121160 18650 ts_tablet_manager.cc:1428] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 10:47:21.121603 18633 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
I20260924 10:47:21.122109 18652 raft_consensus.cc:348] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.122355 18652 raft_consensus.cc:374] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:21.122419 18652 raft_consensus.cc:729] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b28dc8c6c048c1b66276fd5775f07b, State: Initialized, Role: FOLLOWER
I20260924 10:47:21.122928 18652 consensus_queue.cc:260] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.122851 18651 raft_consensus.cc:348] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.123066 18651 raft_consensus.cc:374] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:21.123140 18651 raft_consensus.cc:729] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e33712d5f412daf6a697d50c98a49, State: Initialized, Role: FOLLOWER
I20260924 10:47:21.123549 18651 consensus_queue.cc:260] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.124017 18652 ts_tablet_manager.cc:1428] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260924 10:47:21.124258 18365 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
I20260924 10:47:21.124406 18651 ts_tablet_manager.cc:1428] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:47:21.125162 18499 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
I20260924 10:47:21.293762 18656 raft_consensus.cc:482] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:21.293963 18656 raft_consensus.cc:504] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
W20260924 10:47:21.294638 18634 tablet.cc:2369] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:21.295269 18656 leader_election.cc:288] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
I20260924 10:47:21.300395 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.295537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.2.182.3:34303)
0924 10:47:21.295955 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.295966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.296154 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.296575 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.296580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.296602 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:21.296748 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.296757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.297623 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.297626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.299268 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.299278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.299934 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.299938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.300040 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.300093 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.300176 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.300248 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":239,"thread_start_us":132,"threads_started":1}
I20260924 10:47:21.300401 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.295642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:34303 (local address 127.2.182.1:45697)
0924 10:47:21.295781 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.295786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.296073 (+ 287us) server_negotiation.cc:407] Connection header received
0924 10:47:21.296289 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.296293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.296358 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.296415 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:21.296887 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.297479 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.299394 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.299792 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.299928 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.299998 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.300202 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.300282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:47:21.301293 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.295537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.2.182.3:56879)
0924 10:47:21.295761 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.295779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:21.295887 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:21.296354 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:21.296357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:21.296377 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:21.296536 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.296545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.297587 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.297591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.299268 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:21.299278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.300862 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.300868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:21.300996 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.301014 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:47:21.301109 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:47:21.301181 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":215}
I20260924 10:47:21.301323 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.295686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:56879 (local address 127.2.182.2:32997)
0924 10:47:21.295828 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.295833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.295850 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:47:21.296062 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.296066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.296132 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.296219 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:21.296688 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.297469 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.300049 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.300623 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.300740 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.300801 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.301128 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.301209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:47:21.301355 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3299587b24102b4d1ca41dce426f2" candidate_uuid: "a4608a43dd814c138371a63fa03079d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75e33712d5f412daf6a697d50c98a49" is_pre_election: true
I20260924 10:47:21.301687 18319 raft_consensus.cc:2454] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4608a43dd814c138371a63fa03079d7 in term 0.
I20260924 10:47:21.302166 18522 leader_election.cc:302] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4608a43dd814c138371a63fa03079d7, f75e33712d5f412daf6a697d50c98a49; no voters:
I20260924 10:47:21.302129 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3299587b24102b4d1ca41dce426f2" candidate_uuid: "a4608a43dd814c138371a63fa03079d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b28dc8c6c048c1b66276fd5775f07b" is_pre_election: true
I20260924 10:47:21.302418 18453 raft_consensus.cc:2454] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4608a43dd814c138371a63fa03079d7 in term 0.
I20260924 10:47:21.302419 18656 raft_consensus.cc:2781] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:47:21.302518 18656 raft_consensus.cc:482] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:47:21.302579 18656 raft_consensus.cc:3037] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:21.303658 18656 raft_consensus.cc:504] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.304133 18656 leader_election.cc:288] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [CANDIDATE]: Term 1 election: Requested vote from peers f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
I20260924 10:47:21.304358 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3299587b24102b4d1ca41dce426f2" candidate_uuid: "a4608a43dd814c138371a63fa03079d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75e33712d5f412daf6a697d50c98a49"
I20260924 10:47:21.304487 18319 raft_consensus.cc:3037] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:21.304468 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3299587b24102b4d1ca41dce426f2" candidate_uuid: "a4608a43dd814c138371a63fa03079d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b28dc8c6c048c1b66276fd5775f07b"
I20260924 10:47:21.304579 18453 raft_consensus.cc:3037] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:21.305581 18319 raft_consensus.cc:2454] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4608a43dd814c138371a63fa03079d7 in term 1.
I20260924 10:47:21.305859 18453 raft_consensus.cc:2454] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4608a43dd814c138371a63fa03079d7 in term 1.
I20260924 10:47:21.305884 18522 leader_election.cc:302] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4608a43dd814c138371a63fa03079d7, f75e33712d5f412daf6a697d50c98a49; no voters:
I20260924 10:47:21.306203 18656 raft_consensus.cc:2781] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:21.306464 18656 raft_consensus.cc:686] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 1 LEADER]: Becoming Leader. State: Replica: a4608a43dd814c138371a63fa03079d7, State: Running, Role: LEADER
I20260924 10:47:21.306824 18656 consensus_queue.cc:237] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:21.313925 18174 catalog_manager.cc:5495] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4608a43dd814c138371a63fa03079d7 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "a4608a43dd814c138371a63fa03079d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } health_report { overall_health: HEALTHY } } }
W20260924 10:47:21.328677 18500 tablet.cc:2369] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:21.330830 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.327389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.2.182.3:44073)
0924 10:47:21.327502 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:21.327507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:21.327526 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:47:21.327616 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:21.327618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:21.327672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:21.327767 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:21.327968 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.328466 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.329567 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:21.329776 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:21.329857 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:21.330091 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:21.330463 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:47:21.330575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:21.330600 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 10:47:21.330647 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:47:21.336190 18453 raft_consensus.cc:1261] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Refusing update from remote peer a4608a43dd814c138371a63fa03079d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:21.336191 18319 raft_consensus.cc:1261] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Refusing update from remote peer a4608a43dd814c138371a63fa03079d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:21.336869 18660 consensus_queue.cc:1035] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:21.337105 18656 consensus_queue.cc:1035] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:21.343679 18665 mvcc.cc:205] Tried to move back new op lower bound from 7332851062106316800 to 7332851061995950080. Current Snapshot: MvccSnapshot[applied={T|T < 7332851062106316800}]
I20260924 10:47:21.345346 18664 mvcc.cc:205] Tried to move back new op lower bound from 7332851062106316800 to 7332851061995950080. Current Snapshot: MvccSnapshot[applied={T|T < 7332851062106316800}]
I20260924 10:47:21.351215 18174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:21.351379 18174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:21.353775 18174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Sending DeleteTablet for 3 replicas of tablet 0af3299587b24102b4d1ca41dce426f2
I20260924 10:47:21.354578 18433 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3299587b24102b4d1ca41dce426f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:21 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:53280
I20260924 10:47:21.354588 18299 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3299587b24102b4d1ca41dce426f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:21 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:55150
I20260924 10:47:21.354763 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18236
I20260924 10:47:21.356480 18567 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3299587b24102b4d1ca41dce426f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:21 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:50636
I20260924 10:47:21.356962 18674 tablet_replica.cc:330] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: stopping tablet replica
I20260924 10:47:21.357263 18674 raft_consensus.cc:2229] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:47:21.357563 18674 raft_consensus.cc:2258] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:21.358691 18674 ts_tablet_manager.cc:1905] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:21.360786 18674 ts_tablet_manager.cc:1918] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:21.360852 18674 log.cc:1196] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/0af3299587b24102b4d1ca41dce426f2
I20260924 10:47:21.361209 18674 ts_tablet_manager.cc:1939] T 0af3299587b24102b4d1ca41dce426f2 P a4608a43dd814c138371a63fa03079d7: Deleting consensus metadata
I20260924 10:47:21.361407 18673 tablet_replica.cc:330] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: stopping tablet replica
I20260924 10:47:21.361771 18673 raft_consensus.cc:2229] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:21.362129 18673 raft_consensus.cc:2258] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:21.362396 18159 catalog_manager.cc:4841] TS a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): tablet 0af3299587b24102b4d1ca41dce426f2 (table test-table [id=ed91b3dad91e452d840988981db7e7c8]) successfully deleted
W20260924 10:47:21.362582 18160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:45697 (error 108)
I20260924 10:47:21.362816 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18370
I20260924 10:47:21.368175 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18504
I20260924 10:47:21.370332 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.370063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.0.0.1:53292)
0924 10:47:21.370227 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.370268 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 10:47:21.370337 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:21.370031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.0.0.1:55152)
0924 10:47:21.370171 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:21.370245 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:47:21.370559 18159 catalog_manager.cc:4584] TS 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0af3299587b24102b4d1ca41dce426f2: Network error: Client connection negotiation failed: client connection to 127.2.182.2:32997: connect: Connection refused (error 111)
I20260924 10:47:21.373221 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18142
I20260924 10:47:41.379523 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--webserver_interface=127.2.182.62
--webserver_port=45133
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:47:41.490670 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:41.490921 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:41.490962 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:41.494927 18676 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:47:41.494973 18676 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:47:41.494989 18676 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:47:41.495009 18676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:47:41.495060 18676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:41.495096 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:41.495116 18676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:47:41.495147 18676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:47:41.500248 18676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:41.501302 18676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:41.502329 18676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:47:41.515801 18676 server_base.cc:1034] running on GCE node
W20260924 10:47:41.515755 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.515755 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.515775 18685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:41.516700 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:41.517895 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:41.519084 18676 hybrid_clock.cc:648] HybridClock initialized: now 1790246861519075 us; error 34 us; skew 500 ppm
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:41.521865 18676 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:47:41.522804 18676 webserver.cc:460] Webserver started at http://127.2.182.62:45133/ using document root <none> and password file <none>
I20260924 10:47:41.523404 18676 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:41.523509 18676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:41.527331 18676 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:47:41.529471 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:41.530771 18676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:41.530876 18676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "07be7f40dc2342c7e2dc84b3d038ff23"
server_key_iv: "a297f6bc07daa7c881c38885b238aed2"
server_key_version: "encryptionkey@0"
I20260924 10:47:41.531435 18676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:41.570948 18676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:41.574092 18676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:41.574255 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:41.581365 18676 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:44635
I20260924 10:47:41.581707 18744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:44635 every 8 connection(s)
I20260924 10:47:41.582494 18676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:47:41.586530 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18676
I20260924 10:47:41.586951 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:45697
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=36827
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:41.587121 18745 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:47:41.589087 18745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap starting.
I20260924 10:47:41.613102 18745 log.cc:824] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:41.623845 18745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:41.624298 18745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap complete.
I20260924 10:47:41.629704 18745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:41.630281 18745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Initialized, Role: FOLLOWER
I20260924 10:47:41.630851 18745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:41.630995 18745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:47:41.631070 18745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:47:41.631170 18745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:47:41.633672 18745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:41.634012 18745 leader_election.cc:302] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96ed11b1cd6408ea2d6d0a71ac5ed25; no voters:
I20260924 10:47:41.634325 18745 leader_election.cc:288] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:47:41.634444 18749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:47:41.634748 18749 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 LEADER]: Becoming Leader. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Running, Role: LEADER
I20260924 10:47:41.635155 18749 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:47:41.635551 18745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:47:41.636153 18749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96ed11b1cd6408ea2d6d0a71ac5ed25. Latest consensus state: current_term: 2 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:47:41.636261 18749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:41.636621 18754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:47:41.636767 18750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:47:41.636971 18750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
I20260924 10:47:41.641268 18754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:47:41.641844 18754 catalog_manager.cc:1259] Loaded cluster ID: ecb78fcb5c9547a4a6315377180ec554
I20260924 10:47:41.641908 18754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:47:41.644263 18754 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:47:41.645020 18754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Loaded TSK: 0
I20260924 10:47:41.657059 18754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Generated new TSK 1
I20260924 10:47:41.657320 18754 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:47:41.717355 18747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:41.717583 18747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:41.717624 18747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:41.721581 18747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:41.721630 18747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:41.721712 18747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:47:41.726835 18747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:45697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=36827
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:41.727764 18747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:41.728711 18747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:41.739024 18773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.739032 18775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.739032 18772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:41.739815 18747 server_base.cc:1034] running on GCE node
I20260924 10:47:41.740324 18747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:41.741137 18747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:41.742343 18747 hybrid_clock.cc:648] HybridClock initialized: now 1790246861742304 us; error 47 us; skew 500 ppm
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:41.745782 18747 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:47:41.746737 18747 webserver.cc:460] Webserver started at http://127.2.182.1:36827/ using document root <none> and password file <none>
I20260924 10:47:41.747299 18747 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:41.747372 18747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:41.751317 18747 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:47:41.753603 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:41.754850 18747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:41.754940 18747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "f75e33712d5f412daf6a697d50c98a49"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1dcb3cbe5bda0ddd0771363cd7b5686"
server_key_iv: "eb34d7be628c2e8b18f90547e400a1e2"
server_key_version: "encryptionkey@0"
I20260924 10:47:41.755491 18747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:41.777163 18747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:41.780499 18747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:41.780705 18747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:41.781328 18747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:41.782373 18789 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:47:41.786309 18747 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:47:41.786398 18747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 10:47:41.786489 18747 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:47:41.788125 18747 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:47:41.788192 18747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:47:41.788447 18789 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Bootstrap starting.
I20260924 10:47:41.798713 18747 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:45697
I20260924 10:47:41.798813 18896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:45697 every 8 connection(s)
I20260924 10:47:41.799651 18747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:47:41.804194 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18747
I20260924 10:47:41.804733 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:32997
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=37397
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:41.807772 18789 log.cc:824] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:41.817246 18789 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:41.817477 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.801817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:56537 (local address 127.2.182.62:44635)
0924 10:47:41.802201 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:41.802221 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:41.802603 (+ 382us) server_negotiation.cc:407] Connection header received
0924 10:47:41.804108 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:41.804125 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:41.804519 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:41.804871 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:41.806649 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.807949 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.809424 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.809865 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.809957 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.812509 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:41.812514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:41.812525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:41.812550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:41.814263 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:41.814840 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:41.814850 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:41.814854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:41.814963 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:41.815301 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:41.815306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:41.815308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:41.815628 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:41.815769 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:41.816287 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 10:47:41.816537 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
I20260924 10:47:41.817476 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.801812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:56537)
0924 10:47:41.802397 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:41.802470 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:41.803634 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:41.805182 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:41.805191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:41.805765 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:41.806461 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.806480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.808119 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.808122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.809255 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.809264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.810142 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.810145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.810269 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.810875 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:41.810891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:41.812320 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:41.814394 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:41.814400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:41.814411 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:41.814719 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:41.815065 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:41.815070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:41.815072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:41.815208 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:41.815744 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:41.815750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:41.815995 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 10:47:41.816168 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 10:47:41.816524 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":129,"threads_started":1}
I20260924 10:47:41.817673 18789 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Bootstrap complete.
I20260924 10:47:41.819052 18789 ts_tablet_manager.cc:1397] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.007s
I20260924 10:47:41.819970 18897 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:41.820182 18897 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:41.820639 18897 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:41.821900 18789 raft_consensus.cc:348] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:41.822383 18789 raft_consensus.cc:729] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e33712d5f412daf6a697d50c98a49, State: Initialized, Role: FOLLOWER
I20260924 10:47:41.822841 18789 consensus_queue.cc:260] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:41.822935 18709 ts_manager.cc:194] Registered new tserver with Master: f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697)
I20260924 10:47:41.823796 18789 ts_tablet_manager.cc:1428] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 10:47:41.826237 18709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:56537
I20260924 10:47:41.828739 18897 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.1@KRBTEST.COM
I20260924 10:47:41.836143 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.825747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.0.0.1:36930)
0924 10:47:41.826617 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:41.826661 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:41.826781 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:41.827241 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:41.827247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:41.827571 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:41.827660 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.827666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.828579 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.828584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.829419 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.829428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.830345 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.830348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.830456 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.830965 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:41.830980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:41.832377 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:41.834441 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:41.834449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:41.834453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:41.834689 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:41.834976 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:41.834978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:41.834979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:41.835062 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:41.835673 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:41.835677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:41.835769 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:47:41.835879 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:47:41.836024 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":107,"threads_started":1}
I20260924 10:47:41.836326 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.826425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.2.182.1:45697)
0924 10:47:41.826802 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:41.826813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:41.826836 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:47:41.826903 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:41.826910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:41.827096 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:41.827242 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:41.827773 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.828462 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.829616 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.830115 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.830225 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.832509 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:41.832520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:41.832533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:41.832557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:41.834279 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:41.834794 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:41.834800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:41.834802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:41.834888 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:41.835199 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:41.835203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:41.835204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:41.835557 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:41.835691 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:41.835977 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:47:41.836135 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260924 10:47:41.837208 18831 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3299587b24102b4d1ca41dce426f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:21 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36930
I20260924 10:47:41.838097 18906 tablet_replica.cc:330] stopping tablet replica
I20260924 10:47:41.838368 18906 raft_consensus.cc:2229] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:41.838624 18906 raft_consensus.cc:2258] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:41.839501 18906 ts_tablet_manager.cc:1905] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:41.843014 18906 ts_tablet_manager.cc:1918] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:41.843118 18906 log.cc:1196] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal/wals/0af3299587b24102b4d1ca41dce426f2
I20260924 10:47:41.843401 18906 ts_tablet_manager.cc:1939] T 0af3299587b24102b4d1ca41dce426f2 P f75e33712d5f412daf6a697d50c98a49: Deleting consensus metadata
I20260924 10:47:41.844978 18695 catalog_manager.cc:4841] TS f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697): tablet 0af3299587b24102b4d1ca41dce426f2 (table test-table [id=ed91b3dad91e452d840988981db7e7c8]) successfully deleted
W20260924 10:47:41.922260 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:41.922494 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:41.922533 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:41.926530 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:41.926584 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:41.926666 18901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:47:41.931774 18901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:32997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=37397
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:41.932749 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:41.933744 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:41.943261 18912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.943315 18914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:41.943272 18911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:41.943598 18901 server_base.cc:1034] running on GCE node
I20260924 10:47:41.944008 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:41.944590 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:41.945775 18901 hybrid_clock.cc:648] HybridClock initialized: now 1790246861945759 us; error 34 us; skew 500 ppm
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:41.948381 18901 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:47:41.949254 18901 webserver.cc:460] Webserver started at http://127.2.182.2:37397/ using document root <none> and password file <none>
I20260924 10:47:41.949815 18901 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:41.949918 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:41.953737 18901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:47:41.955781 18921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:41.957098 18901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:47:41.957219 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "63b28dc8c6c048c1b66276fd5775f07b"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1d090fda72fe7bea9b34689b5016e9f"
server_key_iv: "53d6deb09b288b068ea8029531464de1"
server_key_version: "encryptionkey@0"
I20260924 10:47:41.957762 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:41.966667 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:41.969319 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:41.969518 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:41.970049 18901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:41.971100 18928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:47:41.975051 18901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:47:41.975121 18901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 10:47:41.975193 18901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:47:41.976719 18901 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:47:41.976783 18901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:47:41.977099 18928 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Bootstrap starting.
I20260924 10:47:41.987109 18901 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:32997
I20260924 10:47:41.987164 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:32997 every 8 connection(s)
I20260924 10:47:41.988049 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:47:41.992224 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 18901
I20260924 10:47:41.992712 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:44073
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=43317
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:47:41.996876 18928 log.cc:824] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:47:41 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:42.003590 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.989998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:47217 (local address 127.2.182.62:44635)
0924 10:47:41.990147 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:41.990150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:41.990902 (+ 752us) server_negotiation.cc:407] Connection header received
0924 10:47:41.991817 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:41.991820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:41.991881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:41.991984 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:41.993317 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.993862 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.995577 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.995900 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.996033 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.999118 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:41.999122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:41.999125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:41.999154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:42.001031 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.001724 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.001728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.001730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.001820 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.002261 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.002264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.002266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.002488 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:42.002591 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.003292 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.003430 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:47:42.004786 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:41.990235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:47217)
0924 10:47:41.990751 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:41.990791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:41.991611 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:41.992140 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:41.992154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:41.992577 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:41.993173 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.993188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.994020 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.994023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.995423 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:41.995437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:41.996031 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:41.996034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:41.996181 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:41.996819 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:41.996834 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:41.998900 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:42.001167 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:42.001175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:42.001195 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:42.001606 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:42.001936 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:42.001943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:42.001948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:42.002149 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:42.002610 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:42.002620 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:42.002977 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.003238 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.003637 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":162,"threads_started":1}
I20260924 10:47:42.005069 18928 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:47:42.005520 18928 tablet_bootstrap.cc:492] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Bootstrap complete.
I20260924 10:47:42.006119 19036 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:42.006347 19036 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:42.006726 18928 ts_tablet_manager.cc:1397] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.011s
I20260924 10:47:42.006889 19036 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:42.009002 18709 ts_manager.cc:194] Registered new tserver with Master: 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
I20260924 10:47:42.009479 18928 raft_consensus.cc:348] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:42.009954 18928 raft_consensus.cc:729] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b28dc8c6c048c1b66276fd5775f07b, State: Initialized, Role: FOLLOWER
I20260924 10:47:42.010294 18709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:47217
I20260924 10:47:42.010438 18928 consensus_queue.cc:260] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } } peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } }
I20260924 10:47:42.011473 18928 ts_tablet_manager.cc:1428] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:47:42.012765 19036 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
I20260924 10:47:42.016505 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.009977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.0.0.1:32990)
0924 10:47:42.010265 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.010279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:42.010356 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:42.012190 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:42.012193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:42.012213 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:42.012308 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.012314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.013745 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.013749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.015083 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.015095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.016160 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.016163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.016242 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.016283 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.016343 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.016407 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260924 10:47:42.016614 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.010620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.2.182.2:32997)
0924 10:47:42.011349 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.011360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.011381 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:47:42.011445 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.011450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.012001 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.012183 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:42.012428 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.013611 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.015235 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.016040 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.016147 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.016282 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.016382 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.016469 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":108,"threads_started":1}
I20260924 10:47:42.017671 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3299587b24102b4d1ca41dce426f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:21 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:32990
I20260924 10:47:42.018469 19043 tablet_replica.cc:330] stopping tablet replica
I20260924 10:47:42.018788 19043 raft_consensus.cc:2229] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:42.019021 19043 raft_consensus.cc:2258] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:42.020035 19043 ts_tablet_manager.cc:1905] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:42.024348 19043 ts_tablet_manager.cc:1918] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:42.024442 19043 log.cc:1196] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/0af3299587b24102b4d1ca41dce426f2
I20260924 10:47:42.024725 19043 ts_tablet_manager.cc:1939] T 0af3299587b24102b4d1ca41dce426f2 P 63b28dc8c6c048c1b66276fd5775f07b: Deleting consensus metadata
I20260924 10:47:42.026697 18694 catalog_manager.cc:4841] TS 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): tablet 0af3299587b24102b4d1ca41dce426f2 (table test-table [id=ed91b3dad91e452d840988981db7e7c8]) successfully deleted
W20260924 10:47:42.111651 19039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:47:42.111882 19039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:47:42.111922 19039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:47:42.115942 19039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:47:42.116000 19039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:47:42.116097 19039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:47:42.121223 19039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:44073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=43317
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:47:42.122166 19039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:47:42.123143 19039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:47:42.132354 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:47:42.132395 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:42.132411 19039 server_base.cc:1034] running on GCE node
W20260924 10:47:42.132366 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:47:42.132999 19039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:47:42.133526 19039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:47:42.134738 19039 hybrid_clock.cc:648] HybridClock initialized: now 1790246862134726 us; error 27 us; skew 500 ppm
Sep 24 10:47:42 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:47:42.137677 19039 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:47:42.138484 19039 webserver.cc:460] Webserver started at http://127.2.182.3:43317/ using document root <none> and password file <none>
I20260924 10:47:42.139029 19039 fs_manager.cc:359] Metadata directory not provided
I20260924 10:47:42.139151 19039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:47:42.142886 19039 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:47:42.144773 19058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:42.145991 19039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:47:42.146124 19039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "a4608a43dd814c138371a63fa03079d7"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a4f762acbe41c80cf306d3b3beffdaa9"
server_key_iv: "51ac588156fc69dc20ac90a482398ed1"
server_key_version: "encryptionkey@0"
I20260924 10:47:42.146634 19039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:47:42.161525 19039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:47:42.164297 19039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:47:42.164487 19039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:47:42.165093 19039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:47:42.165870 19039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:47:42.165940 19039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:42.166009 19039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:47:42.166060 19039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:47:42.175230 19039 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44073
I20260924 10:47:42.175264 19171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44073 every 8 connection(s)
I20260924 10:47:42.176187 19039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:47:42.180596 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19039
Sep 24 10:47:42 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:47:42.189378 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.181297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.2.182.62:44635)
0924 10:47:42.181530 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.181533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.181683 (+ 150us) server_negotiation.cc:407] Connection header received
0924 10:47:42.181848 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.181850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.181898 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.182008 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:42.182771 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.183309 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.184175 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.184396 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.184505 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.185957 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:42.185960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:42.185963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:42.185986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:42.187703 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.188137 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.188142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.188144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.188225 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.188496 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.188502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.188507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.188680 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:42.188897 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.189066 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.189184 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260924 10:47:42.190599 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.177999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:53595 (local address 127.2.182.62:44635)
0924 10:47:42.178314 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.178317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.178934 (+ 617us) server_negotiation.cc:407] Connection header received
0924 10:47:42.180303 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.180307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.180367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.180500 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:42.182010 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.182627 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.183821 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.184062 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.184184 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.186813 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:42.186818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:42.186821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:42.186866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:42.188216 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.188795 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.188801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.188803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.188944 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.189445 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.189449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.189452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.189676 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:42.189777 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.190369 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.190494 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":235}
I20260924 10:47:42.191562 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.178273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:53595)
0924 10:47:42.178745 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.178800 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:42.180084 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:42.180663 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:42.180670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:42.181166 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:47:42.181838 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.181859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.182765 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.182769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.183686 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.183696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.184218 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.184222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.184410 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.185097 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:47:42.185124 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:47:42.186646 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:47:42.188343 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:42.188350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:42.188395 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:42.188681 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:42.189046 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:47:42.189049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:47:42.189051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:47:42.189194 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:47:42.189778 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:47:42.189786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:47:42.190094 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.190314 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.190654 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260924 10:47:42.192586 19172 heartbeater.cc:347] Connected to a master server at 127.2.182.62:44635
I20260924 10:47:42.192819 19172 heartbeater.cc:464] Registering TS with master...
I20260924 10:47:42.193295 19172 heartbeater.cc:511] Master 127.2.182.62:44635 requested a full tablet report, sending...
I20260924 10:47:42.194248 18709 ts_manager.cc:194] Registered new tserver with Master: a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073)
I20260924 10:47:42.194789 18709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:53595
I20260924 10:47:42.202885 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:47:42.206573 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.203533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.2.182.62:44635)
0924 10:47:42.203666 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.203670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.203772 (+ 102us) server_negotiation.cc:407] Connection header received
0924 10:47:42.203897 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.203899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.203999 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.204087 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:42.204251 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.204725 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.205576 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.205777 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.205882 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.206174 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:42.206289 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 10:47:42.206486 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38462: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 10:47:42.206653 18899 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38462: token expired
W20260924 10:47:42.206939 18636 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.182.62:44635)
I20260924 10:47:42.213869 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.207328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.2.182.62:44635)
0924 10:47:42.207431 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.207436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.207458 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:47:42.207527 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.207529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.207561 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.207650 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:47:42.208171 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.208632 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.209451 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.209647 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.209741 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.210876 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:47:42.210879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:47:42.210881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:47:42.210901 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:47:42.212425 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.212825 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.212828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.212829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.212959 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:47:42.213203 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:47:42.213205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:47:42.213206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:47:42.213359 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:47:42.213469 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.213693 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.213779 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:47:42.215983 18709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:47:42.219161 18709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:47:42.225958 18831 tablet_service.cc:1467] Processing CreateTablet for tablet 4469dfc28f594711892640afe416ee3c (DEFAULT_TABLE table=test-table [id=835fc901f2ed44a9a81a2cdb5f9af283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:42.225958 18970 tablet_service.cc:1467] Processing CreateTablet for tablet 4469dfc28f594711892640afe416ee3c (DEFAULT_TABLE table=test-table [id=835fc901f2ed44a9a81a2cdb5f9af283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:42.226537 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4469dfc28f594711892640afe416ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:42.226537 18831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4469dfc28f594711892640afe416ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:47:42.229266 18928 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Bootstrap starting.
I20260924 10:47:42.229475 18789 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Bootstrap starting.
I20260924 10:47:42.229751 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.225226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44073 (local address 127.0.0.1:44136)
0924 10:47:42.225379 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.225393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:42.225455 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:42.226235 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:42.226238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:42.226255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:42.226328 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.226332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.227541 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.227548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.228464 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.228480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.229421 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.229424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.229505 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.229526 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.229577 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.229643 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 10:47:42.230160 18928 tablet_bootstrap.cc:654] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:42.230311 18789 tablet_bootstrap.cc:654] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:42.230883 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.225377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.2.182.3:44073)
0924 10:47:42.225803 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.225811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.225830 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:47:42.225907 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.225910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.226067 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.226450 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:42.226606 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.227373 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.228591 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.229273 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.230008 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.230194 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.230246 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.230634 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":132,"threads_started":1}
I20260924 10:47:42.231602 18928 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: No bootstrap required, opened a new log
I20260924 10:47:42.231685 18928 ts_tablet_manager.cc:1397] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 10:47:42.231696 18789 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: No bootstrap required, opened a new log
I20260924 10:47:42.231770 18789 ts_tablet_manager.cc:1397] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 10:47:42.232057 18928 raft_consensus.cc:348] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.232170 18928 raft_consensus.cc:374] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:42.232158 18789 raft_consensus.cc:348] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.232216 18928 raft_consensus.cc:729] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b28dc8c6c048c1b66276fd5775f07b, State: Initialized, Role: FOLLOWER
I20260924 10:47:42.232255 18789 raft_consensus.cc:374] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:42.232299 18789 raft_consensus.cc:729] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e33712d5f412daf6a697d50c98a49, State: Initialized, Role: FOLLOWER
I20260924 10:47:42.232337 18928 consensus_queue.cc:260] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.232417 18789 consensus_queue.cc:260] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.232661 18789 ts_tablet_manager.cc:1428] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:47:42.233296 18928 ts_tablet_manager.cc:1428] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:47:42.233953 19106 tablet_service.cc:1467] Processing CreateTablet for tablet 4469dfc28f594711892640afe416ee3c (DEFAULT_TABLE table=test-table [id=835fc901f2ed44a9a81a2cdb5f9af283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:47:42.234915 19106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4469dfc28f594711892640afe416ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 10:47:42.239061 19037 tablet.cc:2369] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:42.241717 19184 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Bootstrap starting.
I20260924 10:47:42.243822 19184 tablet_bootstrap.cc:654] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Neither blocks nor log segments found. Creating new log.
I20260924 10:47:42.244843 19184 log.cc:824] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:47:42.246526 19184 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: No bootstrap required, opened a new log
I20260924 10:47:42.246723 19184 ts_tablet_manager.cc:1397] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 10:47:42.249871 19184 raft_consensus.cc:348] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.250133 19184 raft_consensus.cc:374] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:47:42.250212 19184 raft_consensus.cc:729] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4608a43dd814c138371a63fa03079d7, State: Initialized, Role: FOLLOWER
I20260924 10:47:42.250694 19184 consensus_queue.cc:260] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.251317 19172 heartbeater.cc:503] Master 127.2.182.62:44635 was elected leader, sending a full tablet report...
I20260924 10:47:42.251636 19184 ts_tablet_manager.cc:1428] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260924 10:47:42.301021 18898 tablet.cc:2369] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:42.422602 19042 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:47:42.422798 19042 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.423897 19042 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697)
W20260924 10:47:42.427102 19173 tablet.cc:2369] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:47:42.427489 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.424221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:41587 (local address 127.2.182.3:44073)
0924 10:47:42.424354 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.424359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.424376 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:47:42.424455 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.424461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.424519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.424641 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:42.424953 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.425454 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.426463 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.426857 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.426970 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.427034 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.427313 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.427383 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:47:42.427497 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.424058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44073 (local address 127.2.182.2:41587)
0924 10:47:42.424231 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.424249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:42.424352 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:42.424656 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:42.424659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:42.424680 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:42.424799 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.424805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.425573 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.425576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.426339 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.426347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.427052 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.427056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.427165 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.427193 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.427273 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.427346 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"spinlock_wait_cycles":5376}
I20260924 10:47:42.428284 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.424205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.2.182.2:42407)
0924 10:47:42.424572 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.424585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:47:42.424716 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:47:42.425109 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:47:42.425112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:47:42.425129 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:47:42.425257 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.425263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.426045 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.426051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.426854 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:47:42.426863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.427985 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.427988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:47:42.428062 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.428082 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:47:42.428125 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 10:47:42.428174 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"spinlock_wait_cycles":12544,"thread_start_us":112,"threads_started":1}
I20260924 10:47:42.428231 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4469dfc28f594711892640afe416ee3c" candidate_uuid: "63b28dc8c6c048c1b66276fd5775f07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4608a43dd814c138371a63fa03079d7" is_pre_election: true
I20260924 10:47:42.428367 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.424341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:42407 (local address 127.2.182.1:45697)
0924 10:47:42.424638 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.424645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.424665 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:47:42.424847 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.424852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.424978 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.425100 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:47:42.425379 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.425922 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.426996 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.427727 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.427934 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.428096 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.428164 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.428243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260924 10:47:42.428544 19126 raft_consensus.cc:2454] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b28dc8c6c048c1b66276fd5775f07b in term 0.
I20260924 10:47:42.428993 18851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4469dfc28f594711892640afe416ee3c" candidate_uuid: "63b28dc8c6c048c1b66276fd5775f07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75e33712d5f412daf6a697d50c98a49" is_pre_election: true
I20260924 10:47:42.429078 18923 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7; no voters:
I20260924 10:47:42.429270 18851 raft_consensus.cc:2454] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b28dc8c6c048c1b66276fd5775f07b in term 0.
I20260924 10:47:42.429368 19042 raft_consensus.cc:2781] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:47:42.429451 19042 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:47:42.429514 19042 raft_consensus.cc:3037] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:42.430575 19042 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.430951 19042 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [CANDIDATE]: Term 1 election: Requested vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697)
I20260924 10:47:42.431277 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4469dfc28f594711892640afe416ee3c" candidate_uuid: "63b28dc8c6c048c1b66276fd5775f07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4608a43dd814c138371a63fa03079d7"
I20260924 10:47:42.431306 18851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4469dfc28f594711892640afe416ee3c" candidate_uuid: "63b28dc8c6c048c1b66276fd5775f07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75e33712d5f412daf6a697d50c98a49"
I20260924 10:47:42.431404 19126 raft_consensus.cc:3037] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:42.431404 18851 raft_consensus.cc:3037] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:47:42.432587 18851 raft_consensus.cc:2454] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b28dc8c6c048c1b66276fd5775f07b in term 1.
I20260924 10:47:42.432600 19126 raft_consensus.cc:2454] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b28dc8c6c048c1b66276fd5775f07b in term 1.
I20260924 10:47:42.432869 18923 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7; no voters:
I20260924 10:47:42.433080 19042 raft_consensus.cc:2781] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:47:42.433323 19042 raft_consensus.cc:686] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 1 LEADER]: Becoming Leader. State: Replica: 63b28dc8c6c048c1b66276fd5775f07b, State: Running, Role: LEADER
I20260924 10:47:42.433575 19042 consensus_queue.cc:237] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:47:42.436332 18706 catalog_manager.cc:5495] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b reported cstate change: term changed from 0 to 1, leader changed from <none> to 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2). New cstate: current_term: 1 leader_uuid: "63b28dc8c6c048c1b66276fd5775f07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } health_report { overall_health: HEALTHY } } }
I20260924 10:47:42.465480 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.461847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.2.182.2:32997)
0924 10:47:42.461972 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:47:42.461975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:47:42.461989 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:47:42.462093 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:47:42.462095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:47:42.462148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:47:42.462270 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:47:42.462547 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.463037 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.464068 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:47:42.464294 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:47:42.464382 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:47:42.464626 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:47:42.465148 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:47:42.465250 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:47:42.465314 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:47:42.465364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:47:42.469381 18851 raft_consensus.cc:1261] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Refusing update from remote peer 63b28dc8c6c048c1b66276fd5775f07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:42.469383 19126 raft_consensus.cc:1261] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Refusing update from remote peer 63b28dc8c6c048c1b66276fd5775f07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:47:42.470069 19189 consensus_queue.cc:1035] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:42.470257 19042 consensus_queue.cc:1035] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:47:42.476529 19193 mvcc.cc:205] Tried to move back new op lower bound from 7332851148667817984 to 7332851148530221056. Current Snapshot: MvccSnapshot[applied={T|T < 7332851148667817984}]
I20260924 10:47:42.483456 18708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:42.483633 18708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:47:42.486079 18708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Sending DeleteTablet for 3 replicas of tablet 4469dfc28f594711892640afe416ee3c
I20260924 10:47:42.486779 18831 tablet_service.cc:1514] Processing DeleteTablet for tablet 4469dfc28f594711892640afe416ee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:42 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36930
I20260924 10:47:42.486863 19106 tablet_service.cc:1514] Processing DeleteTablet for tablet 4469dfc28f594711892640afe416ee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:42 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:44136
I20260924 10:47:42.486929 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18747
I20260924 10:47:42.489583 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 4469dfc28f594711892640afe416ee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:47:42 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:32990
I20260924 10:47:42.489679 19202 tablet_replica.cc:330] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: stopping tablet replica
I20260924 10:47:42.490029 19202 raft_consensus.cc:2229] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:47:42.490338 19202 raft_consensus.cc:2258] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:42.490628 19043 tablet_replica.cc:330] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: stopping tablet replica
I20260924 10:47:42.490758 19043 raft_consensus.cc:2229] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:47:42.491055 19043 raft_consensus.cc:2258] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:47:42.491842 19043 ts_tablet_manager.cc:1905] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:47:42.492312 19202 ts_tablet_manager.cc:1905] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:47:42.494613 18695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:45697 (error 108)
I20260924 10:47:42.494876 19202 ts_tablet_manager.cc:1918] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:47:42.494971 19202 log.cc:1196] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/4469dfc28f594711892640afe416ee3c
I20260924 10:47:42.495324 19202 ts_tablet_manager.cc:1939] T 4469dfc28f594711892640afe416ee3c P a4608a43dd814c138371a63fa03079d7: Deleting consensus metadata
I20260924 10:47:42.495468 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18901
I20260924 10:47:42.496605 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.496239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:45697 (local address 127.0.0.1:36944)
0924 10:47:42.496427 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.496509 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:47:42.496851 18694 catalog_manager.cc:4841] TS a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): tablet 4469dfc28f594711892640afe416ee3c (table test-table [id=835fc901f2ed44a9a81a2cdb5f9af283]) successfully deleted
W20260924 10:47:42.496881 18695 catalog_manager.cc:4584] TS f75e33712d5f412daf6a697d50c98a49 (127.2.182.1:45697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4469dfc28f594711892640afe416ee3c: Network error: Client connection negotiation failed: client connection to 127.2.182.1:45697: connect: Connection refused (error 111)
I20260924 10:47:42.501572 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19039
I20260924 10:47:42.503255 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:47:42.501774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.0.0.1:33004)
0924 10:47:42.503076 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0924 10:47:42.503137 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1205}
I20260924 10:47:42.507037 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 18676
I20260924 10:48:02.514349 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--webserver_interface=127.2.182.62
--webserver_port=45133
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:02.618724 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:02.618955 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:02.619025 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:02.622925 19204 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:48:02.622965 19204 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:48:02.622982 19204 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:48:02.622994 19204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:02.623039 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:02.623073 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:02.623100 19204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:02.623165 19204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:02.628022 19204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:02.628979 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:02.629942 19204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:02.637624 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:02.637751 19204 server_base.cc:1034] running on GCE node
W20260924 10:48:02.637624 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:02.637616 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:02.638391 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:02.639431 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:02.640586 19204 hybrid_clock.cc:648] HybridClock initialized: now 1790246882640576 us; error 23 us; skew 500 ppm
Sep 24 10:48:02 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:02.643415 19204 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:02.644281 19204 webserver.cc:460] Webserver started at http://127.2.182.62:45133/ using document root <none> and password file <none>
I20260924 10:48:02.644770 19204 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:02.644827 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:02.648383 19204 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:02.650414 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:02.651542 19204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:02.651631 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "07be7f40dc2342c7e2dc84b3d038ff23"
server_key_iv: "a297f6bc07daa7c881c38885b238aed2"
server_key_version: "encryptionkey@0"
I20260924 10:48:02.652094 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:02.685766 19204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:02.688748 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:02.688920 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:02.695458 19204 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:44635
I20260924 10:48:02.695736 19271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:44635 every 8 connection(s)
I20260924 10:48:02.696486 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:02.700340 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19204
I20260924 10:48:02.700572 19272 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:48:02.700838 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:45697
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=36827
--webserver_interface=127.2.182.1
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:02.702310 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap starting.
I20260924 10:48:02.726183 19272 log.cc:824] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:02.737879 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:48:02.738251 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Bootstrap complete.
I20260924 10:48:02.742244 19272 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:48:02.742635 19272 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Initialized, Role: FOLLOWER
I20260924 10:48:02.743129 19272 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:48:02.743240 19272 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:02.743326 19272 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:02.743443 19272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:48:02.745357 19272 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:48:02.745689 19272 leader_election.cc:302] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96ed11b1cd6408ea2d6d0a71ac5ed25; no voters:
I20260924 10:48:02.746021 19272 leader_election.cc:288] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:48:02.746102 19276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:48:02.746323 19276 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [term 3 LEADER]: Becoming Leader. State: Replica: b96ed11b1cd6408ea2d6d0a71ac5ed25, State: Running, Role: LEADER
I20260924 10:48:02.746677 19276 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } }
I20260924 10:48:02.747233 19272 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:02.747649 19276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96ed11b1cd6408ea2d6d0a71ac5ed25. Latest consensus state: current_term: 3 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:48:02.747768 19276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:02.748143 19280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:02.748201 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96ed11b1cd6408ea2d6d0a71ac5ed25" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 44635 } } }
I20260924 10:48:02.748382 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:02.753108 19280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:02.753962 19280 catalog_manager.cc:1259] Loaded cluster ID: ecb78fcb5c9547a4a6315377180ec554
I20260924 10:48:02.754024 19280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:02.756467 19280 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:02.757205 19280 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Loaded TSK: 0
I20260924 10:48:02.757283 19280 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b96ed11b1cd6408ea2d6d0a71ac5ed25: Loaded TSK: 1
F20260924 10:48:02.758006 19280 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f523d0119ad google::LogMessage::Fail() at ??:0
@ 0x7f523d01488a google::LogMessage::SendToLog() at ??:0
@ 0x7f523d0114de google::LogMessage::Flush() at ??:0
@ 0x7f523d012a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f523fcd518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f523fccdf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f52479767bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f524795099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5247951178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f5247992df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f52479b182b std::function<>::operator()() at ??:0
@ 0x7f5247950b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5247951f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f524794e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5247992640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f523f2633d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f523f263c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f523f2653f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f523f254512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f52408986db start_thread at ??:0
@ 0x7f523bb0f61f clone at ??:0
W20260924 10:48:02.837576 19274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:02.837849 19274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:02.837895 19274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:02.842132 19274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:02.842173 19274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:02.842231 19274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:02.847755 19274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:45697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=36827
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:02.848719 19274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:02.849967 19274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:02.859644 19319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:02.859644 19320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:02.859902 19274 server_base.cc:1034] running on GCE node
W20260924 10:48:02.860086 19322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:02.860602 19274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:02.861430 19274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:02.862630 19274 hybrid_clock.cc:648] HybridClock initialized: now 1790246882862590 us; error 49 us; skew 500 ppm
Sep 24 10:48:02 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:02.866674 19274 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:48:02.867796 19274 webserver.cc:460] Webserver started at http://127.2.182.1:36827/ using document root <none> and password file <none>
I20260924 10:48:02.868372 19274 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:02.868465 19274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:02.872447 19274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:48:02.874666 19329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:02.875934 19274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:02.876063 19274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "f75e33712d5f412daf6a697d50c98a49"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1dcb3cbe5bda0ddd0771363cd7b5686"
server_key_iv: "eb34d7be628c2e8b18f90547e400a1e2"
server_key_version: "encryptionkey@0"
I20260924 10:48:02.876583 19274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:02.903896 19274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:02.907150 19274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:02.907342 19274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:02.907908 19274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:02.909052 19336 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:48:02.914314 19274 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:48:02.914386 19274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 10:48:02.914484 19274 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:48:02.916093 19274 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:48:02.916162 19274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:48:02.916468 19336 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Bootstrap starting.
I20260924 10:48:02.926715 19274 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:45697
I20260924 10:48:02.926841 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:45697 every 8 connection(s)
I20260924 10:48:02.927731 19274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:02.929224 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19274
I20260924 10:48:02.929678 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:32997
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=37397
--webserver_interface=127.2.182.2
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:02.938552 19336 log.cc:824] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:02.947691 19336 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:48:02.948110 19336 tablet_bootstrap.cc:492] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Bootstrap complete.
I20260924 10:48:02.949573 19336 ts_tablet_manager.cc:1397] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.011s
I20260924 10:48:02.952875 19336 raft_consensus.cc:348] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:02.953410 19336 raft_consensus.cc:729] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e33712d5f412daf6a697d50c98a49, State: Initialized, Role: FOLLOWER
I20260924 10:48:02.953976 19336 consensus_queue.cc:260] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:02.955037 19336 ts_tablet_manager.cc:1428] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260924 10:48:02.998354 19446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:02.930403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:42755)
0924 10:48:02.930890 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:02.930944 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:02.932083 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:02.997436 (+ 65353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":140,"threads_started":1}
W20260924 10:48:02.998905 19444 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:44635 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:44635: Client connection negotiation failed: client connection to 127.2.182.62:44635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:48:02.999723 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:02.999400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:33147)
0924 10:48:02.999556 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:02.999644 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:48:03.000391 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.000170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:44549)
0924 10:48:03.000288 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.000338 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:48:03.000590 19444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:48:03.039793 19447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:03.040023 19447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:03.040095 19447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:03.043983 19447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:03.044041 19447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:03.044132 19447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:03.049132 19447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:32997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=37397
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:03.050065 19447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:03.051039 19447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:03.060199 19454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:03.060182 19455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:03.060235 19457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:03.060534 19447 server_base.cc:1034] running on GCE node
I20260924 10:48:03.061010 19447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:03.061585 19447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:03.062736 19447 hybrid_clock.cc:648] HybridClock initialized: now 1790246883062706 us; error 41 us; skew 500 ppm
Sep 24 10:48:03 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:03.065343 19447 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:48:03.066201 19447 webserver.cc:460] Webserver started at http://127.2.182.2:37397/ using document root <none> and password file <none>
I20260924 10:48:03.066735 19447 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:03.066795 19447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:03.070487 19447 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:03.072629 19464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:03.073889 19447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:03.074013 19447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "63b28dc8c6c048c1b66276fd5775f07b"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a1d090fda72fe7bea9b34689b5016e9f"
server_key_iv: "53d6deb09b288b068ea8029531464de1"
server_key_version: "encryptionkey@0"
I20260924 10:48:03.074528 19447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:03.099711 19447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:03.102607 19447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:03.102798 19447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:03.103395 19447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:03.104442 19471 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 10:48:03.108282 19471 ts_tablet_manager.cc:1852] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 10:48:03.108680 19471 ts_tablet_manager.cc:1905] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:03.112848 19471 ts_tablet_manager.cc:1918] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:03.113018 19471 log.cc:1196] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/4469dfc28f594711892640afe416ee3c
I20260924 10:48:03.113389 19471 ts_tablet_manager.cc:1939] T 4469dfc28f594711892640afe416ee3c P 63b28dc8c6c048c1b66276fd5775f07b: Deleting consensus metadata
I20260924 10:48:03.113710 19447 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 10:48:03.113782 19447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260924 10:48:03.113875 19447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:03.113925 19447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:03.122644 19447 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:32997
I20260924 10:48:03.122680 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:32997 every 8 connection(s)
I20260924 10:48:03.123472 19447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:03.126717 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19447
I20260924 10:48:03.126724 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.125530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:42119)
0924 10:48:03.125947 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.126059 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260924 10:48:03.127183 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:44073
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:44635
--webserver_port=43317
--webserver_interface=127.2.182.3
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:03.127255 19579 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:44635 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:44635: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
I20260924 10:48:03.127868 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.127652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:39223)
0924 10:48:03.127762 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.127809 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:48:03.128636 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.128370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:53613)
0924 10:48:03.128503 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.128564 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 10:48:03.128814 19579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:48:03.236706 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:03.236953 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:03.236994 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:03.240841 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:03.240904 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:03.240960 19582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:03.245941 19582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:44073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=43317
--tserver_master_addrs=127.2.182.62:44635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:03.246901 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:03.247846 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:03.257146 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:03.257146 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:03.257386 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:03.257602 19582 server_base.cc:1034] running on GCE node
I20260924 10:48:03.258000 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:03.258610 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:03.259791 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790246883259774 us; error 27 us; skew 500 ppm
Sep 24 10:48:03 dist-test-slave-6lwt krb5kdc[18129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:03.262473 19582 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:48:03.263347 19582 webserver.cc:460] Webserver started at http://127.2.182.3:43317/ using document root <none> and password file <none>
I20260924 10:48:03.263900 19582 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:03.263990 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:03.267709 19582 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:03.269779 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:03.270967 19582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:03.271103 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "a4608a43dd814c138371a63fa03079d7"
format_stamp: "Formatted at 2026-09-24 10:47:20 on dist-test-slave-6lwt"
server_key: "a4f762acbe41c80cf306d3b3beffdaa9"
server_key_iv: "51ac588156fc69dc20ac90a482398ed1"
server_key_version: "encryptionkey@0"
I20260924 10:48:03.271713 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:03.281829 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:03.284482 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:03.284693 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:03.285300 19582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:03.286095 19582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:03.286164 19582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:03.286230 19582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:03.286279 19582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:03.295104 19582 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44073
I20260924 10:48:03.295132 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:44073 every 8 connection(s)
I20260924 10:48:03.296048 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:48:03.299641 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.298044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:39661)
0924 10:48:03.298472 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.298961 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
W20260924 10:48:03.300113 19711 heartbeater.cc:644] Failed to heartbeat to 127.2.182.62:44635 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.62:44635: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
I20260924 10:48:03.300642 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.300453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:54189)
0924 10:48:03.300549 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.300591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:48:03.301209 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:03.301039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:59983)
0924 10:48:03.301127 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:03.301166 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 10:48:03.301385 19711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:48:03.303436 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19582
I20260924 10:48:04.001577 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.001138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:48943)
0924 10:48:04.001408 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:04.001466 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260924 10:48:04.129570 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.129240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:45747)
0924 10:48:04.129444 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:04.129492 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260924 10:48:04.302220 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.301850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:56301)
0924 10:48:04.302074 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:04.302125 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
I20260924 10:48:04.670657 19718 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:04.670912 19718 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:04.672021 19718 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
Sep 24 10:48:04 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.3@KRBTEST.COM
Sep 24 10:48:04 dist-test-slave-6lwt krb5kdc[18129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.2@KRBTEST.COM
I20260924 10:48:04.693806 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.672747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:32997 (local address 127.2.182.1:39455)
0924 10:48:04.673108 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:04.673122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:04.673287 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:04.674693 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:04.674696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:04.675237 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:04.675749 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:04.675759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.677365 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.677369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:04.678317 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:04.678344 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.680227 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.680231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:04.680318 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:04.686959 (+ 6641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:04.686986 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:04.689142 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:04.691345 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:04.691352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:04.691365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:04.691650 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:04.692014 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:04.692020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:04.692025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:04.692188 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:04.692824 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:04.692828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:04.693071 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 10:48:04.693257 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:48:04.693628 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":170,"spinlock_wait_cycles":47744,"thread_start_us":84,"threads_started":1}
I20260924 10:48:04.693818 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.672493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:34119 (local address 127.2.182.3:44073)
0924 10:48:04.672868 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:04.672918 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:04.673133 (+ 215us) server_negotiation.cc:407] Connection header received
0924 10:48:04.674013 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:04.674018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:04.674342 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:04.674627 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:04.675853 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.677180 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.678456 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.679010 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.679175 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:04.689393 (+ 10218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:04.689398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:04.689409 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:04.689436 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:04.691337 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:04.691833 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:04.691837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:04.691841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:04.691939 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:04.692322 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:04.692324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:04.692325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:04.692677 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:04.692806 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:04.693363 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 10:48:04.693642 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260924 10:48:04.693820 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.672913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:39455 (local address 127.2.182.2:32997)
0924 10:48:04.673233 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:04.673248 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:04.673289 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:48:04.674063 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:04.674069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:04.674351 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:04.674634 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:04.675954 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.677216 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.679337 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.680090 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.680256 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:04.689331 (+ 9075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:04.689337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:04.689349 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:04.689378 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:04.691227 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:04.691760 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:04.691768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:04.691775 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:04.691905 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:04.692283 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:04.692287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:04.692289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:04.692670 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:04.692833 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:04.693362 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:48:04.693641 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260924 10:48:04.693863 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:04.672330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:44073 (local address 127.2.182.1:34119)
0924 10:48:04.673005 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:04.673018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:04.673237 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:04.674679 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:04.674685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:04.675310 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:04.675715 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:04.675730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.677345 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.677348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:04.678314 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:04.678322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:04.679138 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:04.679141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:04.679317 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:04.686908 (+ 7591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:04.686931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:04.688800 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:04.691449 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:04.691451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:04.691453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:04.691729 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:04.692080 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:04.692082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:04.692083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:04.692196 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:04.692795 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:04.692801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:04.693073 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 10:48:04.693257 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:48:04.693710 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":198,"spinlock_wait_cycles":6016,"thread_start_us":78,"threads_started":1}
W20260924 10:48:04.696635 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
W20260924 10:48:04.696801 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
I20260924 10:48:04.696844 19331 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75e33712d5f412daf6a697d50c98a49; no voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7
I20260924 10:48:04.697126 19718 raft_consensus.cc:2726] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:48:05.002708 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:05.002332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:37531)
0924 10:48:05.002574 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:05.002629 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 10:48:05.130662 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:05.130244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:47469)
0924 10:48:05.130502 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:05.130558 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260924 10:48:05.303373 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:05.302981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:33671)
0924 10:48:05.303238 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:05.303290 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
I20260924 10:48:06.004058 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:06.003605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:33821)
0924 10:48:06.003892 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:06.003948 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260924 10:48:06.131759 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:06.131344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:47603)
0924 10:48:06.131610 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:06.131659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260924 10:48:06.304549 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:06.304121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:53751)
0924 10:48:06.304375 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:06.304426 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260924 10:48:06.370062 19728 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:06.370189 19728 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:06.370667 19728 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
W20260924 10:48:06.371553 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
W20260924 10:48:06.371711 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
I20260924 10:48:06.371773 19331 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75e33712d5f412daf6a697d50c98a49; no voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7
I20260924 10:48:06.371942 19728 raft_consensus.cc:2726] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:48:07.005278 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:07.004835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:54917)
0924 10:48:07.005133 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:07.005185 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":142,"threads_started":1}
I20260924 10:48:07.132836 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:07.132450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:49021)
0924 10:48:07.132683 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:07.132735 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260924 10:48:07.305577 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:07.305234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:55707)
0924 10:48:07.305451 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:07.305501 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260924 10:48:08.006671 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:08.006265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:43327)
0924 10:48:08.006531 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:08.006580 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":68,"threads_started":1}
I20260924 10:48:08.134001 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:08.133589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:41065)
0924 10:48:08.133853 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:08.133908 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260924 10:48:08.306710 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:08.306300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:47289)
0924 10:48:08.306582 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:08.306630 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260924 10:48:09.007876 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:09.007409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:42869)
0924 10:48:09.007703 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:09.007766 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260924 10:48:09.135051 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:09.134689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:42803)
0924 10:48:09.134895 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:09.134945 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260924 10:48:09.307848 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:09.307425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:44695)
0924 10:48:09.307680 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:09.307754 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
I20260924 10:48:09.483142 19738 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:09.483295 19738 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:09.483719 19738 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
W20260924 10:48:09.484560 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
W20260924 10:48:09.484694 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
I20260924 10:48:09.484754 19331 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75e33712d5f412daf6a697d50c98a49; no voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7
I20260924 10:48:09.484972 19738 raft_consensus.cc:2726] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:48:10.009126 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:10.008637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:50597)
0924 10:48:10.008972 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:10.009032 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260924 10:48:10.136171 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:10.135758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:36029)
0924 10:48:10.136035 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:10.136088 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
I20260924 10:48:10.309031 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:10.308496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:54105)
0924 10:48:10.308759 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:10.308808 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
I20260924 10:48:11.010366 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:11.009958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:37035)
0924 10:48:11.010210 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:11.010268 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260924 10:48:11.137326 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:11.136867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:52693)
0924 10:48:11.137188 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:11.137240 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260924 10:48:11.310178 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:11.309752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:56257)
0924 10:48:11.310035 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:11.310090 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260924 10:48:12.011607 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:12.011167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:59473)
0924 10:48:12.011442 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:12.011506 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260924 10:48:12.138562 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:12.138128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:59637)
0924 10:48:12.138406 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:12.138465 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260924 10:48:12.311285 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:12.310900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:54903)
0924 10:48:12.311145 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:12.311197 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260924 10:48:13.012842 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:13.012457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:39149)
0924 10:48:13.012680 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:13.012730 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260924 10:48:13.122170 19749 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:13.122298 19749 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:13.122692 19749 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
W20260924 10:48:13.123493 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
W20260924 10:48:13.123641 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
I20260924 10:48:13.123696 19331 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75e33712d5f412daf6a697d50c98a49; no voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7
I20260924 10:48:13.123831 19749 raft_consensus.cc:2726] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:48:13.139794 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:13.139343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:34153)
0924 10:48:13.139598 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:13.139649 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260924 10:48:13.312423 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:13.312063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:32827)
0924 10:48:13.312290 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:13.312339 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260924 10:48:14.013917 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:14.013516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:47627)
0924 10:48:14.013774 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:14.013825 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260924 10:48:14.140990 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:14.140502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:57241)
0924 10:48:14.140787 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:14.140856 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260924 10:48:14.313623 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:14.313270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:43835)
0924 10:48:14.313490 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:14.313539 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260924 10:48:15.014999 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:15.014624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:40085)
0924 10:48:15.014860 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:15.014908 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260924 10:48:15.142045 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:15.141662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:38439)
0924 10:48:15.141890 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:15.141953 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260924 10:48:15.314630 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:15.314281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:33343)
0924 10:48:15.314498 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:15.314547 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260924 10:48:15.481257 19758 raft_consensus.cc:482] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:15.481383 19758 raft_consensus.cc:504] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4608a43dd814c138371a63fa03079d7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44073 } } peers { permanent_uuid: "f75e33712d5f412daf6a697d50c98a49" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 45697 } } peers { permanent_uuid: "63b28dc8c6c048c1b66276fd5775f07b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32997 } }
I20260924 10:48:15.481834 19758 leader_election.cc:288] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073), 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997)
W20260924 10:48:15.482650 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4608a43dd814c138371a63fa03079d7 (127.2.182.3:44073): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
W20260924 10:48:15.482839 19331 leader_election.cc:341] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63b28dc8c6c048c1b66276fd5775f07b (127.2.182.2:32997): Not found: Tablet not found: 4469dfc28f594711892640afe416ee3c
I20260924 10:48:15.482903 19331 leader_election.cc:302] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75e33712d5f412daf6a697d50c98a49; no voters: 63b28dc8c6c048c1b66276fd5775f07b, a4608a43dd814c138371a63fa03079d7
I20260924 10:48:15.483075 19758 raft_consensus.cc:2726] T 4469dfc28f594711892640afe416ee3c P f75e33712d5f412daf6a697d50c98a49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:48:16.016239 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:16.015855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:42099)
0924 10:48:16.016092 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:16.016145 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260924 10:48:16.143136 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:16.142747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:38469)
0924 10:48:16.142970 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:16.143024 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260924 10:48:16.315717 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:16.315311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:51785)
0924 10:48:16.315586 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:16.315634 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":73,"threads_started":1}
I20260924 10:48:17.017333 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:17.016974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:60113)
0924 10:48:17.017193 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:17.017243 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260924 10:48:17.144198 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:17.143800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:57173)
0924 10:48:17.144054 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:17.144107 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260924 10:48:17.316857 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:17.316485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:57633)
0924 10:48:17.316719 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:17.316770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260924 10:48:18.018657 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:18.018243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.1:53047)
0924 10:48:18.018517 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:18.018567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260924 10:48:18.145578 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:18.145096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.2:58207)
0924 10:48:18.145351 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:18.145407 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260924 10:48:18.307597 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19274
I20260924 10:48:18.313614 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19447
I20260924 10:48:18.318194 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:18.317823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:44635 (local address 127.2.182.3:50975)
0924 10:48:18.318063 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:18.318112 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.62:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260924 10:48:18.319231 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19582
2026-09-24T10:48:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59333 ms)
[----------] 30 tests from SecurityITest (235718 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:18 dist-test-slave-6lwt krb5kdc[19770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:18 dist-test-slave-6lwt krb5kdc[19770](info): set up 2 sockets
Sep 24 10:48:18 dist-test-slave-6lwt krb5kdc[19770](info): commencing operation
krb5kdc: starting...
W20260924 10:48:19.345147 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:19 dist-test-slave-6lwt krb5kdc[19770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:19.479552 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41033
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:19.585457 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:19.585680 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:19.585716 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:19.589591 19783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:19.589634 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:19.589653 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:19.589711 19783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:19.589752 19783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:19.594630 19783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:19.595498 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:19.596441 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:19.604683 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:19.604812 19783 server_base.cc:1034] running on GCE node
W20260924 10:48:19.604722 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:19.604712 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:19.605512 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:19.606539 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:19.607714 19783 hybrid_clock.cc:648] HybridClock initialized: now 1790246899607677 us; error 29 us; skew 500 ppm
Sep 24 10:48:19 dist-test-slave-6lwt krb5kdc[19770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246899, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:19.610333 19783 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:19.611200 19783 webserver.cc:460] Webserver started at http://127.2.182.62:39261/ using document root <none> and password file <none>
I20260924 10:48:19.611732 19783 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:19.611820 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:19.612040 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:19.614055 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "df942333801cd3d78dfe071d7ecd6a25"
server_key_iv: "47d54e1ec9b0a773d5c6c184f7b4b163"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.614591 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "df942333801cd3d78dfe071d7ecd6a25"
server_key_iv: "47d54e1ec9b0a773d5c6c184f7b4b163"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.618076 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:19.620268 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:19.621454 19783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:19.621591 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "df942333801cd3d78dfe071d7ecd6a25"
server_key_iv: "47d54e1ec9b0a773d5c6c184f7b4b163"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.621695 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:19.632700 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:19.635250 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:19.635438 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:19.642294 19783 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:41033
I20260924 10:48:19.642334 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:41033 every 8 connection(s)
I20260924 10:48:19.643182 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:19.645159 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19783
I20260924 10:48:19.645284 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:19.645594 2776 external_mini_cluster.cc:1428] Setting key f5be0919aa36f9fda7d42d3754e7400f
I20260924 10:48:19.646121 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:19.651636 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: Bootstrap starting.
Sep 24 10:48:19 dist-test-slave-6lwt krb5kdc[19770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:19.654366 19851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:19.655273 19851 log.cc:824] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:19.657466 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: No bootstrap required, opened a new log
I20260924 10:48:19.659452 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:19.646923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.2.182.62:41033)
0924 10:48:19.647328 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:19.647342 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:19.647375 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:48:19.648001 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:19.648015 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:19.648336 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:19.648654 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:19.649450 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.650399 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.651265 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.651732 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.651894 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:19.654642 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:19.654649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:19.654669 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:19.654698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:19.656528 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:19.656953 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:19.656959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:19.656964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:19.657092 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:19.657350 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:19.657353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:19.657355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:19.657844 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:19.658021 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:19.658239 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:48:19.658536 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":139,"threads_started":1}
I20260924 10:48:19.661773 19851 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } }
I20260924 10:48:19.662005 19851 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:19.662047 19851 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33ed76a4c3e4d46abe7c0ae0bddd03b, State: Initialized, Role: FOLLOWER
I20260924 10:48:19.662393 19851 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } }
I20260924 10:48:19.662499 19851 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:19.662544 19851 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:19.662611 19851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:19.663431 19851 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } }
I20260924 10:48:19.663708 19851 leader_election.cc:302] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33ed76a4c3e4d46abe7c0ae0bddd03b; no voters:
I20260924 10:48:19.663938 19851 leader_election.cc:288] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:19.664067 19856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:19.664314 19856 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [term 1 LEADER]: Becoming Leader. State: Replica: f33ed76a4c3e4d46abe7c0ae0bddd03b, State: Running, Role: LEADER
I20260924 10:48:19.664624 19856 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } }
I20260924 10:48:19.665179 19851 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:19.666368 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } } }
I20260924 10:48:19.666380 19858 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33ed76a4c3e4d46abe7c0ae0bddd03b. Latest consensus state: current_term: 1 leader_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33ed76a4c3e4d46abe7c0ae0bddd03b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41033 } } }
I20260924 10:48:19.666519 19858 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:19.666518 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:19.666949 19865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:19.670020 19865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:19.675719 19865 catalog_manager.cc:1347] Generated new cluster ID: 9f595150ad6a4265b68b06de700a455f
I20260924 10:48:19.675828 19865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:19.703979 19865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:19.705267 19865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:19.710022 19865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: Generated new TSK 0
I20260924 10:48:19.710803 19865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:48:19.748165 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:19.855866 19877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:19.856146 19877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:19.856242 19877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:19.860283 19877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:19.860368 19877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:19.860453 19877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:19.865653 19877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:19.866696 19877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:19.867717 19877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:19.876899 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:19.876971 19877 server_base.cc:1034] running on GCE node
W20260924 10:48:19.876989 19885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:19.876948 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:19.877558 19877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:19.878095 19877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:19.879262 19877 hybrid_clock.cc:648] HybridClock initialized: now 1790246899879252 us; error 32 us; skew 500 ppm
Sep 24 10:48:19 dist-test-slave-6lwt krb5kdc[19770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246899, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:19.881907 19877 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:48:19.882704 19877 webserver.cc:460] Webserver started at http://127.2.182.1:41697/ using document root <none> and password file <none>
I20260924 10:48:19.883255 19877 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:19.883345 19877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:19.883584 19877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:19.885576 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "9d0b01cecd9d475ab118fb4288b39755"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "d8bf1be5860010aca60fa6a141dc6e40"
server_key_iv: "aebe32879c6328fcf218013d5176ac8b"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.886041 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "9d0b01cecd9d475ab118fb4288b39755"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "d8bf1be5860010aca60fa6a141dc6e40"
server_key_iv: "aebe32879c6328fcf218013d5176ac8b"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.889596 19877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:19.891746 19892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:19.892825 19877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:19.892963 19877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "9d0b01cecd9d475ab118fb4288b39755"
format_stamp: "Formatted at 2026-09-24 10:48:19 on dist-test-slave-6lwt"
server_key: "d8bf1be5860010aca60fa6a141dc6e40"
server_key_iv: "aebe32879c6328fcf218013d5176ac8b"
server_key_version: "encryptionkey@0"
I20260924 10:48:19.893047 19877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:19.912897 19877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:19.915493 19877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:19.915640 19877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:19.916121 19877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:19.916880 19877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:19.916983 19877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:19.917044 19877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:19.917075 19877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:19.925375 19877 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:34933
I20260924 10:48:19.925390 20005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:34933 every 8 connection(s)
I20260924 10:48:19.926239 19877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:48:19 dist-test-slave-6lwt krb5kdc[19770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246899, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:19.934569 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 19877
I20260924 10:48:19.934722 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:19.935003 2776 external_mini_cluster.cc:1428] Setting key f29531cfac2a3a868c258c8b6bf6446a
I20260924 10:48:19.938684 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:19.927940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:58027 (local address 127.2.182.62:41033)
0924 10:48:19.928068 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:19.928072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:19.928776 (+ 704us) server_negotiation.cc:407] Connection header received
0924 10:48:19.929695 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:19.929698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:19.929757 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:19.929855 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:19.930963 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.931504 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.932530 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.932746 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.932852 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:19.935100 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:19.935104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:19.935107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:19.935135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:19.936613 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:19.937181 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:19.937185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:19.937187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:19.937268 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:19.937624 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:19.937627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:19.937629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:19.937804 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:19.937912 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:19.938390 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:48:19.938517 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:19.939447 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:19.928230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:41033 (local address 127.2.182.1:58027)
0924 10:48:19.928640 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:19.928679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:19.929488 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:19.930030 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:19.930036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:19.930384 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:19.930818 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:19.930831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.931618 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.931622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:19.932412 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:19.932421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:19.932886 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:19.932910 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:19.933074 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:19.933541 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:19.933555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:19.934944 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:19.936731 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:19.936737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:19.936750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:19.937077 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:19.937369 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:19.937372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:19.937374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:19.937519 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:19.937909 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:19.937915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:19.938157 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 10:48:19.938334 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 10:48:19.938654 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260924 10:48:19.940651 20006 heartbeater.cc:347] Connected to a master server at 127.2.182.62:41033
I20260924 10:48:19.940912 20006 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:19.941504 20006 heartbeater.cc:511] Master 127.2.182.62:41033 requested a full tablet report, sending...
I20260924 10:48:19.943228 19815 ts_manager.cc:194] Registered new tserver with Master: 9d0b01cecd9d475ab118fb4288b39755 (127.2.182.1:34933)
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:48:19.944893 19815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:58027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:48:19.961503 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:20.068935 20011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:20.069182 20011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:20.069226 20011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:20.073084 20011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:20.073141 20011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:20.073226 20011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:20.078193 20011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:20.079140 20011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:20.080117 20011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:20.089342 20017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:20.089447 20011 server_base.cc:1034] running on GCE node
W20260924 10:48:20.089326 20016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:20.089326 20019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:20.090003 20011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:20.090595 20011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:20.091777 20011 hybrid_clock.cc:648] HybridClock initialized: now 1790246900091747 us; error 44 us; skew 500 ppm
Sep 24 10:48:20 dist-test-slave-6lwt krb5kdc[19770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246900, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:20.094358 20011 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:48:20.095119 20011 webserver.cc:460] Webserver started at http://127.2.182.2:38279/ using document root <none> and password file <none>
I20260924 10:48:20.095691 20011 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:20.095780 20011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:20.095999 20011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:20.098088 20011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "acf723ff9f5f42ff8df784afaf2426f5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "f368ec2eb8cf4c83f35ff1167a8d635a"
server_key_iv: "9d148a461ae99010866f6937d7779603"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.098641 20011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "acf723ff9f5f42ff8df784afaf2426f5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "f368ec2eb8cf4c83f35ff1167a8d635a"
server_key_iv: "9d148a461ae99010866f6937d7779603"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.102144 20011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:20.104414 20026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.105543 20011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:20.105667 20011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "acf723ff9f5f42ff8df784afaf2426f5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "f368ec2eb8cf4c83f35ff1167a8d635a"
server_key_iv: "9d148a461ae99010866f6937d7779603"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.105770 20011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:20.121196 20011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:20.124130 20011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:20.124356 20011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:20.125064 20011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:20.125991 20011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:20.126063 20011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.126137 20011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:20.126191 20011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.134971 20011 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42329
I20260924 10:48:20.134994 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:42329 every 8 connection(s)
I20260924 10:48:20.135818 20011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:20.137112 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20011
I20260924 10:48:20.137254 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:20.137538 2776 external_mini_cluster.cc:1428] Setting key d942c60492e566a9d975db3c50a74970
Sep 24 10:48:20 dist-test-slave-6lwt krb5kdc[19770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246900, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:48:20.151093 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.137824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:35439 (local address 127.2.182.62:41033)
0924 10:48:20.137981 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.137986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.138877 (+ 891us) server_negotiation.cc:407] Connection header received
0924 10:48:20.140038 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.140041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.140088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.140181 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:20.141681 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.142239 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.144051 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.144271 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.144378 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.147369 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:20.147373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:20.147378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:20.147412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:20.149140 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.149611 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.149613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.149615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.149689 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.149997 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.150001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.150003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.150203 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:20.150312 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.150866 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.150970 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
I20260924 10:48:20.151966 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.138238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:41033 (local address 127.2.182.2:35439)
0924 10:48:20.138735 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.138773 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.139790 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.140361 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.140373 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.140930 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:20.141535 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.141547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.142381 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.142386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.143912 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.143927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.144429 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.144436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.144650 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.145339 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:20.145366 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:20.147182 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:20.149251 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:20.149256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:20.149268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:20.149516 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:20.149769 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:20.149771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:20.149772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:20.149896 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:20.150311 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:20.150318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:20.150560 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.150796 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.151061 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260924 10:48:20.153204 20140 heartbeater.cc:347] Connected to a master server at 127.2.182.62:41033
I20260924 10:48:20.153435 20140 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:20.153914 20140 heartbeater.cc:511] Master 127.2.182.62:41033 requested a full tablet report, sending...
I20260924 10:48:20.154963 19815 ts_manager.cc:194] Registered new tserver with Master: acf723ff9f5f42ff8df784afaf2426f5 (127.2.182.2:42329)
I20260924 10:48:20.155537 19815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:35439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:48:20.167152 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:20.274142 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:20.274369 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:20.274408 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:20.278278 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:20.278326 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:20.278379 20145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:20.283341 20145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:20.284260 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:20.285264 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:20.294525 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:20.294548 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:20.294643 20150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:20.294677 20145 server_base.cc:1034] running on GCE node
I20260924 10:48:20.295177 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:20.295660 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:20.296804 20145 hybrid_clock.cc:648] HybridClock initialized: now 1790246900296775 us; error 40 us; skew 500 ppm
Sep 24 10:48:20 dist-test-slave-6lwt krb5kdc[19770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246900, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:20.299367 20145 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:48:20.300176 20145 webserver.cc:460] Webserver started at http://127.2.182.3:39587/ using document root <none> and password file <none>
I20260924 10:48:20.300710 20145 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:20.300767 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:20.301038 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:20.302929 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "73cd3578ce4a48228a2ff1ef37913fd5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "514ca5e7c04e57b2643dab4b86eb7a4d"
server_key_iv: "9a68663522951bb741264fc90590601d"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.303479 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "73cd3578ce4a48228a2ff1ef37913fd5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "514ca5e7c04e57b2643dab4b86eb7a4d"
server_key_iv: "9a68663522951bb741264fc90590601d"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.306898 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:20.309157 20160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.310354 20145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:48:20.310487 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "73cd3578ce4a48228a2ff1ef37913fd5"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "514ca5e7c04e57b2643dab4b86eb7a4d"
server_key_iv: "9a68663522951bb741264fc90590601d"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.310581 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:20.321679 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:20.324225 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:20.324427 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:20.324878 20145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:20.325659 20145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:20.325701 20145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.325733 20145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:20.325754 20145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.334092 20145 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:46539
I20260924 10:48:20.334127 20273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:46539 every 8 connection(s)
I20260924 10:48:20.334851 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:48:20 dist-test-slave-6lwt krb5kdc[19770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246900, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:20.343276 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20145
I20260924 10:48:20.343432 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:20.343708 2776 external_mini_cluster.cc:1428] Setting key 7b668fcdea647d984e178161acc15067
I20260924 10:48:20.347455 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.336426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:34723 (local address 127.2.182.62:41033)
0924 10:48:20.336558 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.336562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.337341 (+ 779us) server_negotiation.cc:407] Connection header received
0924 10:48:20.338227 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.338231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.338287 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.338362 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:20.339398 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.339913 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.341105 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.341328 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.341439 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.343855 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:20.343858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:20.343860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:20.343883 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:20.345420 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.346005 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.346009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.346011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.346095 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.346437 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.346441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.346442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.346630 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:20.346732 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.347172 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.347286 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:48:20.348145 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.336751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:41033 (local address 127.2.182.3:34723)
0924 10:48:20.337211 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.337249 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.338030 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.338515 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.338521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.338845 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:20.339262 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.339273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.340044 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.340047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.340982 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.340999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.341468 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.341472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.341641 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.342174 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:20.342188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:20.343675 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:20.345528 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:20.345533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:20.345545 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:20.345898 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:20.346185 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:20.346189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:20.346191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:20.346322 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:20.346730 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:20.346736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:20.346960 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.347133 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.347407 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":164,"threads_started":1}
I20260924 10:48:20.349170 20274 heartbeater.cc:347] Connected to a master server at 127.2.182.62:41033
I20260924 10:48:20.349387 20274 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:20.349810 20274 heartbeater.cc:511] Master 127.2.182.62:41033 requested a full tablet report, sending...
I20260924 10:48:20.350719 19815 ts_manager.cc:194] Registered new tserver with Master: 73cd3578ce4a48228a2ff1ef37913fd5 (127.2.182.3:46539)
I20260924 10:48:20.351254 19815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:34723
I20260924 10:48:20.357477 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:20.366204 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.2.182.62:41033)
0924 10:48:20.359380 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.359383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.359560 (+ 177us) server_negotiation.cc:407] Connection header received
0924 10:48:20.359646 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.359648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.359698 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.359762 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:20.360492 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.361022 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.361811 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.362001 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.362070 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.363233 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:20.363235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:20.363237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:20.363257 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:20.364605 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.365002 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.365004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.365005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.365082 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:20.365366 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:20.365368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:20.365370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:20.365562 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:20.365711 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.365939 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.366037 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:20.368652 19815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:20.371449 19815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:20.383781 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.378620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:46539 (local address 127.0.0.1:58504)
0924 10:48:20.378938 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.378980 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.379083 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.379669 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.379674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.379696 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:20.379804 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.379811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.381074 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.381077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.382286 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.382296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.383274 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.383277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.383374 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.383408 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.383540 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.383649 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":31,"thread_start_us":71,"threads_started":1}
I20260924 10:48:20.383786 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.378748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.2.182.3:46539)
0924 10:48:20.379076 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.379085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.379106 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:48:20.379175 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.379181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.379364 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.379503 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:20.379919 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.380877 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.382425 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.383162 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.383274 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.383402 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.383566 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.383649 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":130,"threads_started":1}
I20260924 10:48:20.384459 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.379237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42329 (local address 127.0.0.1:34582)
0924 10:48:20.379871 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.379883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.379953 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.380382 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.380385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.380402 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:20.380482 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.380488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.381802 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.381805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.382878 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.382887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.384055 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.384059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.384161 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.384182 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.384249 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.384329 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":32,"thread_start_us":77,"threads_started":1}
I20260924 10:48:20.384876 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.379412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.2.182.2:42329)
0924 10:48:20.379996 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.380010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.380031 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:48:20.380097 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.380102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.380259 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.380367 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:20.380612 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.381672 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.382996 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.383782 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.383872 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.383993 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.384635 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.384733 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260924 10:48:20.385365 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.379528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:34933 (local address 127.0.0.1:45666)
0924 10:48:20.381065 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.381078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.381187 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.381560 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.381562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.381577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:20.381655 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.381661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.382600 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.382602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.383692 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.383703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.384986 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.384989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.385087 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.385104 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.385188 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.385252 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":69,"threads_started":1}
I20260924 10:48:20.385720 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.379685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.2.182.1:34933)
0924 10:48:20.380638 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.380652 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.381181 (+ 529us) server_negotiation.cc:407] Connection header received
0924 10:48:20.381294 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.381299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.381446 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.381568 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:20.381738 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.382473 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.383831 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.384685 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.384776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.384895 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.385233 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.385565 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":78,"threads_started":1}
I20260924 10:48:20.386967 20208 tablet_service.cc:1467] Processing CreateTablet for tablet 72509efff32e4a1fae62af9cb7a1049b (DEFAULT_TABLE table=test-table [id=ac293aa093e2457d845de2e768477d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:20.387265 20074 tablet_service.cc:1467] Processing CreateTablet for tablet 72509efff32e4a1fae62af9cb7a1049b (DEFAULT_TABLE table=test-table [id=ac293aa093e2457d845de2e768477d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:20.387984 20208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72509efff32e4a1fae62af9cb7a1049b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:20.388165 20074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72509efff32e4a1fae62af9cb7a1049b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:20.387970 19940 tablet_service.cc:1467] Processing CreateTablet for tablet 72509efff32e4a1fae62af9cb7a1049b (DEFAULT_TABLE table=test-table [id=ac293aa093e2457d845de2e768477d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:20.388796 19940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72509efff32e4a1fae62af9cb7a1049b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:20.393616 20291 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Bootstrap starting.
I20260924 10:48:20.395254 20293 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Bootstrap starting.
I20260924 10:48:20.395872 20292 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Bootstrap starting.
I20260924 10:48:20.396384 20291 tablet_bootstrap.cc:654] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:20.397161 20291 log.cc:824] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:20.397194 20293 tablet_bootstrap.cc:654] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:20.397873 20292 tablet_bootstrap.cc:654] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:20.398361 20293 log.cc:824] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:20.398962 20291 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: No bootstrap required, opened a new log
I20260924 10:48:20.398962 20292 log.cc:824] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:20.399147 20291 ts_tablet_manager.cc:1397] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:48:20.400235 20293 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: No bootstrap required, opened a new log
I20260924 10:48:20.400472 20293 ts_tablet_manager.cc:1397] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:20.400830 20292 tablet_bootstrap.cc:492] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: No bootstrap required, opened a new log
I20260924 10:48:20.401024 20292 ts_tablet_manager.cc:1397] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:20.402247 20291 raft_consensus.cc:348] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.402441 20291 raft_consensus.cc:374] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:20.402475 20291 raft_consensus.cc:729] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf723ff9f5f42ff8df784afaf2426f5, State: Initialized, Role: FOLLOWER
I20260924 10:48:20.402781 20291 consensus_queue.cc:260] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.403330 20140 heartbeater.cc:503] Master 127.2.182.62:41033 was elected leader, sending a full tablet report...
I20260924 10:48:20.403687 20291 ts_tablet_manager.cc:1428] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 10:48:20.404707 20293 raft_consensus.cc:348] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.404707 20292 raft_consensus.cc:348] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.404955 20292 raft_consensus.cc:374] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:20.405016 20292 raft_consensus.cc:729] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cd3578ce4a48228a2ff1ef37913fd5, State: Initialized, Role: FOLLOWER
I20260924 10:48:20.405050 20293 raft_consensus.cc:374] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:20.405174 20293 raft_consensus.cc:729] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0b01cecd9d475ab118fb4288b39755, State: Initialized, Role: FOLLOWER
I20260924 10:48:20.405453 20292 consensus_queue.cc:260] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.405670 20293 consensus_queue.cc:260] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.406301 20292 ts_tablet_manager.cc:1428] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 10:48:20.406524 20293 ts_tablet_manager.cc:1428] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:48:20.406643 20274 heartbeater.cc:503] Master 127.2.182.62:41033 was elected leader, sending a full tablet report...
I20260924 10:48:20.406821 20006 heartbeater.cc:503] Master 127.2.182.62:41033 was elected leader, sending a full tablet report...
W20260924 10:48:20.427227 20007 tablet.cc:2369] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:20.545686 20297 raft_consensus.cc:482] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:20.545881 20297 raft_consensus.cc:504] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.546969 20297 leader_election.cc:288] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cd3578ce4a48228a2ff1ef37913fd5 (127.2.182.3:46539), 9d0b01cecd9d475ab118fb4288b39755 (127.2.182.1:34933)
I20260924 10:48:20.550554 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.547350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48883 (local address 127.2.182.3:46539)
0924 10:48:20.547481 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.547485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.547505 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:48:20.547562 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.547565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.547615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.547705 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:20.547995 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.548519 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.549603 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.550001 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.550121 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.550193 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.550367 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.550437 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:48:20.550555 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.547213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:46539 (local address 127.2.182.2:48883)
0924 10:48:20.547346 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.547363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.547457 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.547720 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.547723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.547741 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:20.547870 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.547876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.548632 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.548639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.549469 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.549478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.550120 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.550124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.550230 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.550251 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.550349 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.550426 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":19}
I20260924 10:48:20.551049 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.547488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:34933 (local address 127.2.182.2:60401)
0924 10:48:20.547907 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.547918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:20.548028 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:20.548294 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:20.548297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:20.548311 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:20.548434 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.548439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.549187 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.549192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.549976 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:20.549984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.550608 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.550612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:20.550716 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.550748 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:48:20.550856 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:48:20.550918 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260924 10:48:20.551057 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.547605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:60401 (local address 127.2.182.1:34933)
0924 10:48:20.547760 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.547766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.548005 (+ 239us) server_negotiation.cc:407] Connection header received
0924 10:48:20.548120 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.548125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.548183 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.548283 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:20.548546 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.549084 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.550099 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.550477 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.550598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.550654 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.550878 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.550943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:48:20.551223 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72509efff32e4a1fae62af9cb7a1049b" candidate_uuid: "acf723ff9f5f42ff8df784afaf2426f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" is_pre_election: true
I20260924 10:48:20.551554 20228 raft_consensus.cc:2454] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acf723ff9f5f42ff8df784afaf2426f5 in term 0.
I20260924 10:48:20.551610 19960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72509efff32e4a1fae62af9cb7a1049b" candidate_uuid: "acf723ff9f5f42ff8df784afaf2426f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0b01cecd9d475ab118fb4288b39755" is_pre_election: true
I20260924 10:48:20.551872 19960 raft_consensus.cc:2454] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acf723ff9f5f42ff8df784afaf2426f5 in term 0.
I20260924 10:48:20.552067 20029 leader_election.cc:302] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cd3578ce4a48228a2ff1ef37913fd5, acf723ff9f5f42ff8df784afaf2426f5; no voters:
I20260924 10:48:20.552366 20297 raft_consensus.cc:2781] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:20.552441 20297 raft_consensus.cc:482] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:20.552474 20297 raft_consensus.cc:3037] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:20.553574 20297 raft_consensus.cc:504] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.553887 20297 leader_election.cc:288] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [CANDIDATE]: Term 1 election: Requested vote from peers 73cd3578ce4a48228a2ff1ef37913fd5 (127.2.182.3:46539), 9d0b01cecd9d475ab118fb4288b39755 (127.2.182.1:34933)
I20260924 10:48:20.554227 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72509efff32e4a1fae62af9cb7a1049b" candidate_uuid: "acf723ff9f5f42ff8df784afaf2426f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cd3578ce4a48228a2ff1ef37913fd5"
I20260924 10:48:20.554281 19960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72509efff32e4a1fae62af9cb7a1049b" candidate_uuid: "acf723ff9f5f42ff8df784afaf2426f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0b01cecd9d475ab118fb4288b39755"
I20260924 10:48:20.554337 20228 raft_consensus.cc:3037] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:20.554451 19960 raft_consensus.cc:3037] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:20.555375 20228 raft_consensus.cc:2454] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acf723ff9f5f42ff8df784afaf2426f5 in term 1.
I20260924 10:48:20.555517 19960 raft_consensus.cc:2454] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acf723ff9f5f42ff8df784afaf2426f5 in term 1.
I20260924 10:48:20.555667 20029 leader_election.cc:302] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cd3578ce4a48228a2ff1ef37913fd5, acf723ff9f5f42ff8df784afaf2426f5; no voters:
I20260924 10:48:20.555819 20297 raft_consensus.cc:2781] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:20.556051 20297 raft_consensus.cc:686] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 1 LEADER]: Becoming Leader. State: Replica: acf723ff9f5f42ff8df784afaf2426f5, State: Running, Role: LEADER
I20260924 10:48:20.556381 20297 consensus_queue.cc:237] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } }
I20260924 10:48:20.559502 19815 catalog_manager.cc:5495] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to acf723ff9f5f42ff8df784afaf2426f5 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "acf723ff9f5f42ff8df784afaf2426f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acf723ff9f5f42ff8df784afaf2426f5" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 } health_report { overall_health: UNKNOWN } } }
I20260924 10:48:20.575006 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.572034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.2.182.2:42329)
0924 10:48:20.572158 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.572161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.572172 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:48:20.572303 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.572306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.572349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.572430 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:20.572625 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.573143 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.573895 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.574087 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.574155 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.574387 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:20.574735 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:20.574816 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.574873 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.574910 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:48:20.580111 19960 raft_consensus.cc:1261] T 72509efff32e4a1fae62af9cb7a1049b P 9d0b01cecd9d475ab118fb4288b39755 [term 1 FOLLOWER]: Refusing update from remote peer acf723ff9f5f42ff8df784afaf2426f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:20.580111 20228 raft_consensus.cc:1261] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 1 FOLLOWER]: Refusing update from remote peer acf723ff9f5f42ff8df784afaf2426f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:20.580801 20301 consensus_queue.cc:1035] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cd3578ce4a48228a2ff1ef37913fd5" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:20.581044 20297 consensus_queue.cc:1035] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0b01cecd9d475ab118fb4288b39755" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:48:20.585659 20275 tablet.cc:2369] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:20.588791 20306 mvcc.cc:205] Tried to move back new op lower bound from 7332851304769576960 to 7332851304681500672. Current Snapshot: MvccSnapshot[applied={T|T < 7332851304769576960}]
I20260924 10:48:20.589229 20305 mvcc.cc:205] Tried to move back new op lower bound from 7332851304769576960 to 7332851304681500672. Current Snapshot: MvccSnapshot[applied={T|T < 7332851304769576960}]
I20260924 10:48:20.597643 19815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:20.597800 19815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:20.599887 19815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f33ed76a4c3e4d46abe7c0ae0bddd03b: Sending DeleteTablet for 3 replicas of tablet 72509efff32e4a1fae62af9cb7a1049b
I20260924 10:48:20.600611 20208 tablet_service.cc:1514] Processing DeleteTablet for tablet 72509efff32e4a1fae62af9cb7a1049b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:20 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:58504
I20260924 10:48:20.600647 20074 tablet_service.cc:1514] Processing DeleteTablet for tablet 72509efff32e4a1fae62af9cb7a1049b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:20 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:34582
I20260924 10:48:20.600744 19940 tablet_service.cc:1514] Processing DeleteTablet for tablet 72509efff32e4a1fae62af9cb7a1049b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:20 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260924 10:48:20.601256 20315 tablet_replica.cc:330] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: stopping tablet replica
I20260924 10:48:20.601567 20315 raft_consensus.cc:2229] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:20.601601 20314 tablet_replica.cc:330] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: stopping tablet replica
I20260924 10:48:20.601841 20314 raft_consensus.cc:2229] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:20.601881 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19877
I20260924 10:48:20.601933 20315 raft_consensus.cc:2258] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:20.602788 20314 raft_consensus.cc:2258] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:20.603991 20315 ts_tablet_manager.cc:1905] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:20.606320 20314 ts_tablet_manager.cc:1905] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:20.606576 20315 ts_tablet_manager.cc:1918] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:20.606652 20315 log.cc:1196] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/72509efff32e4a1fae62af9cb7a1049b
I20260924 10:48:20.606961 20315 ts_tablet_manager.cc:1939] T 72509efff32e4a1fae62af9cb7a1049b P acf723ff9f5f42ff8df784afaf2426f5: Deleting consensus metadata
I20260924 10:48:20.608215 19801 catalog_manager.cc:4841] TS acf723ff9f5f42ff8df784afaf2426f5 (127.2.182.2:42329): tablet 72509efff32e4a1fae62af9cb7a1049b (table test-table [id=ac293aa093e2457d845de2e768477d09]) successfully deleted
I20260924 10:48:20.608851 20314 ts_tablet_manager.cc:1918] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:20.608969 20314 log.cc:1196] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/72509efff32e4a1fae62af9cb7a1049b
I20260924 10:48:20.609249 20314 ts_tablet_manager.cc:1939] T 72509efff32e4a1fae62af9cb7a1049b P 73cd3578ce4a48228a2ff1ef37913fd5: Deleting consensus metadata
W20260924 10:48:20.610213 19801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:34933 (error 108)
I20260924 10:48:20.610452 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20011
I20260924 10:48:20.610706 19801 catalog_manager.cc:4841] TS 73cd3578ce4a48228a2ff1ef37913fd5 (127.2.182.3:46539): tablet 72509efff32e4a1fae62af9cb7a1049b (table test-table [id=ac293aa093e2457d845de2e768477d09]) successfully deleted
I20260924 10:48:20.612061 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.611712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:34933 (local address 127.0.0.1:45668)
0924 10:48:20.611865 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:20.611948 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:48:20.612216 19801 catalog_manager.cc:4584] TS 9d0b01cecd9d475ab118fb4288b39755 (127.2.182.1:34933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72509efff32e4a1fae62af9cb7a1049b: Network error: Client connection negotiation failed: client connection to 127.2.182.1:34933: connect: Connection refused (error 111)
I20260924 10:48:20.616294 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20145
I20260924 10:48:20.621429 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 19783
2026-09-24T10:48:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:20Z Disabled control of system clock
I20260924 10:48:20.650575 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38815
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:20.754748 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:20.754977 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:20.755010 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:20.758939 20321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:20.758991 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:20.759006 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:20.759039 20321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:20.759054 20321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:20.763633 20321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:20.764555 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:20.765949 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:20.774227 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:20.774350 20321 server_base.cc:1034] running on GCE node
W20260924 10:48:20.774329 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:20.774519 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:20.775063 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:20.776062 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:20.777211 20321 hybrid_clock.cc:648] HybridClock initialized: now 1790246900777194 us; error 30 us; skew 500 ppm
I20260924 10:48:20.778995 20321 webserver.cc:460] Webserver started at http://127.2.182.62:33045/ using document root <none> and password file <none>
I20260924 10:48:20.779557 20321 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:20.779615 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:20.779830 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:20.781821 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "16f9f0be719044bf90923d7d0bf7951d"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "344c2f14484c996d960550bea97a88cf"
server_key_iv: "42e27348cc349a7ed99f7ea68f626013"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.782333 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "16f9f0be719044bf90923d7d0bf7951d"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "344c2f14484c996d960550bea97a88cf"
server_key_iv: "42e27348cc349a7ed99f7ea68f626013"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.786100 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:20.788445 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:20.789641 20321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:20.789776 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "16f9f0be719044bf90923d7d0bf7951d"
format_stamp: "Formatted at 2026-09-24 10:48:20 on dist-test-slave-6lwt"
server_key: "344c2f14484c996d960550bea97a88cf"
server_key_iv: "42e27348cc349a7ed99f7ea68f626013"
server_key_version: "encryptionkey@0"
I20260924 10:48:20.789875 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:20.819628 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:20.820195 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:20.820396 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:20.826747 20321 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38815
I20260924 10:48:20.826754 20387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:38815 every 8 connection(s)
I20260924 10:48:20.827543 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:20.830400 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:20.835327 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: Bootstrap starting.
I20260924 10:48:20.835893 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20321
I20260924 10:48:20.836017 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:20.836294 2776 external_mini_cluster.cc:1428] Setting key 1e66053e6266b347bc2f7a948350a2e5
I20260924 10:48:20.838052 20388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:20.838730 20388 log.cc:824] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:20.840479 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: No bootstrap required, opened a new log
I20260924 10:48:20.843868 20388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } }
I20260924 10:48:20.844125 20388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:20.844202 20388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f9f0be719044bf90923d7d0bf7951d, State: Initialized, Role: FOLLOWER
I20260924 10:48:20.844703 20388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } }
I20260924 10:48:20.844849 20388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:20.844946 20388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:20.845058 20388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:20.845467 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:20.837590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.2.182.62:38815)
0924 10:48:20.838203 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:20.838216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:20.838249 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:48:20.838840 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:20.838863 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:20.839335 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:20.839662 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:20.840062 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.841106 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.841970 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:20.842484 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:20.842648 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:20.843555 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:20.843561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:20.843575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:20.844024 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:20.844093 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:20.844119 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:20.844222 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:20.844391 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:48:20.844638 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260924 10:48:20.845871 20388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } }
I20260924 10:48:20.846206 20388 leader_election.cc:302] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f9f0be719044bf90923d7d0bf7951d; no voters:
I20260924 10:48:20.846524 20388 leader_election.cc:288] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:20.846580 20393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:20.846740 20393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [term 1 LEADER]: Becoming Leader. State: Replica: 16f9f0be719044bf90923d7d0bf7951d, State: Running, Role: LEADER
I20260924 10:48:20.847002 20393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } }
I20260924 10:48:20.847715 20388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:20.848639 20394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f9f0be719044bf90923d7d0bf7951d. Latest consensus state: current_term: 1 leader_uuid: "16f9f0be719044bf90923d7d0bf7951d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } } }
I20260924 10:48:20.848752 20394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:20.849023 20395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16f9f0be719044bf90923d7d0bf7951d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f9f0be719044bf90923d7d0bf7951d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38815 } } }
I20260924 10:48:20.849112 20395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:20.849352 20402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:20.852063 20402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:20.856798 20402 catalog_manager.cc:1347] Generated new cluster ID: 15bfd951cd1e46ccb7deb9cdb2ac764e
I20260924 10:48:20.856854 20402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:20.869804 20402 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:20.870599 20402 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:20.874006 20402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: Generated new TSK 0
I20260924 10:48:20.874580 20402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:20.878432 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--builtin_ntp_servers=127.2.182.20:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:20.985387 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:20.985632 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:20.985671 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:20.989643 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:20.989692 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:20.989785 20412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:20.994622 20412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:20.995626 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:20.996959 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:21.006680 20418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:21.006714 20417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:21.006944 20412 server_base.cc:1034] running on GCE node
W20260924 10:48:21.006675 20420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:21.007548 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:21.008175 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:21.009332 20412 hybrid_clock.cc:648] HybridClock initialized: now 1790246901009314 us; error 29 us; skew 500 ppm
I20260924 10:48:21.011096 20412 webserver.cc:460] Webserver started at http://127.2.182.1:41915/ using document root <none> and password file <none>
I20260924 10:48:21.011633 20412 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:21.011690 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:21.011850 20412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:21.013929 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "7476ccc0c42440868647f70812a44204"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "047743513379ce1ff00f9aed399e85c0"
server_key_iv: "1158c4f237ba13c2f80f57f23b1811b4"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.014433 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "7476ccc0c42440868647f70812a44204"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "047743513379ce1ff00f9aed399e85c0"
server_key_iv: "1158c4f237ba13c2f80f57f23b1811b4"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.017936 20412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:21.020220 20426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.021481 20412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:21.021612 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "7476ccc0c42440868647f70812a44204"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "047743513379ce1ff00f9aed399e85c0"
server_key_iv: "1158c4f237ba13c2f80f57f23b1811b4"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.021728 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:21.051363 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:21.052081 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:21.052276 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:21.052958 20412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:21.053911 20412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:21.053980 20412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.054049 20412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:21.054090 20412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.062654 20412 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:33691
I20260924 10:48:21.062680 20539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:33691 every 8 connection(s)
I20260924 10:48:21.063529 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:21.064710 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20412
I20260924 10:48:21.064823 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:21.065143 2776 external_mini_cluster.cc:1428] Setting key 2e5d697b1953e435da25b0c713b4afea
I20260924 10:48:21.067071 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--builtin_ntp_servers=127.2.182.20:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:21.074286 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.065294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:54419 (local address 127.2.182.62:38815)
0924 10:48:21.065453 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.065458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.066639 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 10:48:21.068169 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.068172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.068258 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.068398 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:21.069716 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.070241 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.072141 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.072393 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.072478 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.073823 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:21.073826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:21.073828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:21.073844 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:21.073862 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:21.073870 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:21.073936 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.074127 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.074176 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:48:21.075525 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.065784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38815 (local address 127.2.182.1:54419)
0924 10:48:21.066485 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.066548 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.067907 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.068711 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.068722 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.068833 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:21.069539 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.069561 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.070376 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.070383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.071999 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.072013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.072561 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.072569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.072810 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.073515 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:21.073519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:21.073578 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:21.073668 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:21.073958 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:21.073970 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:21.073977 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.074127 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.074450 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260924 10:48:21.076761 20540 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38815
I20260924 10:48:21.077039 20540 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:21.077520 20540 heartbeater.cc:511] Master 127.2.182.62:38815 requested a full tablet report, sending...
I20260924 10:48:21.078939 20352 ts_manager.cc:194] Registered new tserver with Master: 7476ccc0c42440868647f70812a44204 (127.2.182.1:33691)
I20260924 10:48:21.080036 20352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:54419
W20260924 10:48:21.180152 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:21.180565 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:21.180660 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:21.187857 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:21.187944 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:21.188053 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:21.196517 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:21.197647 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:21.198964 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:21.208053 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:21.208144 20543 server_base.cc:1034] running on GCE node
W20260924 10:48:21.208033 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:21.208025 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:21.208745 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:21.209453 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:21.210608 20543 hybrid_clock.cc:648] HybridClock initialized: now 1790246901210591 us; error 31 us; skew 500 ppm
I20260924 10:48:21.212324 20543 webserver.cc:460] Webserver started at http://127.2.182.2:40751/ using document root <none> and password file <none>
I20260924 10:48:21.212982 20543 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:21.213044 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:21.213289 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:21.215217 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "e929e643e4d84f678b0c655efaf31474"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "c589c5dceb4a5574dde962d36ae9aafb"
server_key_iv: "9a622ab4d72835df62774801e678978c"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.215756 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "e929e643e4d84f678b0c655efaf31474"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "c589c5dceb4a5574dde962d36ae9aafb"
server_key_iv: "9a622ab4d72835df62774801e678978c"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.219321 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:21.221555 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.222687 20543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:21.222831 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "e929e643e4d84f678b0c655efaf31474"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "c589c5dceb4a5574dde962d36ae9aafb"
server_key_iv: "9a622ab4d72835df62774801e678978c"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.222967 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:21.243810 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:21.244395 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:21.244567 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:21.245141 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:21.245970 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:21.246035 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.246095 20543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:21.246143 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.255220 20543 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:35343
I20260924 10:48:21.255244 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:35343 every 8 connection(s)
I20260924 10:48:21.256105 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:21.263641 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20543
I20260924 10:48:21.263739 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:21.264107 2776 external_mini_cluster.cc:1428] Setting key efa3eff6c1607f5ef7c348f940c380d1
I20260924 10:48:21.265764 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.257847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:51435 (local address 127.2.182.62:38815)
0924 10:48:21.258021 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.258026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.259163 (+ 1137us) server_negotiation.cc:407] Connection header received
0924 10:48:21.260661 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.260665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.260744 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.260836 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:21.261780 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.262322 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.263538 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.263835 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.263936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.265230 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:21.265235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:21.265238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:21.265250 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:21.265274 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:21.265283 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:21.265360 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.265558 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.265621 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:48:21.266371 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--builtin_ntp_servers=127.2.182.20:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:21.266832 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.258249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38815 (local address 127.2.182.2:51435)
0924 10:48:21.258986 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.259041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.260418 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.261051 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.261058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.261142 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:21.261638 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.261651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.262468 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.262471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.263405 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.263417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.264011 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.264015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.264221 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.264947 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:21.264949 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:21.265004 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:21.265083 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:21.265367 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:21.265374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:21.265378 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.265516 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.265812 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260924 10:48:21.267997 20671 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38815
I20260924 10:48:21.268242 20671 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:21.268850 20671 heartbeater.cc:511] Master 127.2.182.62:38815 requested a full tablet report, sending...
I20260924 10:48:21.269929 20352 ts_manager.cc:194] Registered new tserver with Master: e929e643e4d84f678b0c655efaf31474 (127.2.182.2:35343)
I20260924 10:48:21.270812 20352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:51435
W20260924 10:48:21.380839 20674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:21.381137 20674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:21.381175 20674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:21.385093 20674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:21.385140 20674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:21.385191 20674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:21.390029 20674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:21.390996 20674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:21.392282 20674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:21.401393 20680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:21.401530 20682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:21.401463 20679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:21.401635 20674 server_base.cc:1034] running on GCE node
I20260924 10:48:21.402094 20674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:21.402844 20674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:21.404052 20674 hybrid_clock.cc:648] HybridClock initialized: now 1790246901404050 us; error 36 us; skew 500 ppm
I20260924 10:48:21.406368 20674 webserver.cc:460] Webserver started at http://127.2.182.3:41619/ using document root <none> and password file <none>
I20260924 10:48:21.406929 20674 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:21.406987 20674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:21.407204 20674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:21.409268 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "33f47636c3ec4b2e93f3aae68ebf384c"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "d127b6d6b20642e52ad6912e3a5a301f"
server_key_iv: "61361957b7118c193d49a4c841b2efe6"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.409819 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "33f47636c3ec4b2e93f3aae68ebf384c"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "d127b6d6b20642e52ad6912e3a5a301f"
server_key_iv: "61361957b7118c193d49a4c841b2efe6"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.413403 20674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:21.415696 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.416821 20674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:21.416977 20674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "33f47636c3ec4b2e93f3aae68ebf384c"
format_stamp: "Formatted at 2026-09-24 10:48:21 on dist-test-slave-6lwt"
server_key: "d127b6d6b20642e52ad6912e3a5a301f"
server_key_iv: "61361957b7118c193d49a4c841b2efe6"
server_key_version: "encryptionkey@0"
I20260924 10:48:21.417099 20674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:21.436640 20674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:21.437371 20674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:21.437574 20674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:21.438136 20674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:21.439033 20674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:21.439105 20674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.439175 20674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:21.439213 20674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:21.448474 20674 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:37989
I20260924 10:48:21.448503 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:37989 every 8 connection(s)
I20260924 10:48:21.449482 20674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:48:21.453613 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20674
I20260924 10:48:21.453732 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:21.454036 2776 external_mini_cluster.cc:1428] Setting key fb0d9cfc982c68cf00fcbb0410701a35
I20260924 10:48:21.459023 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.451314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:58025 (local address 127.2.182.62:38815)
0924 10:48:21.451459 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.451464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.452507 (+ 1043us) server_negotiation.cc:407] Connection header received
0924 10:48:21.454110 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.454117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.454179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.454251 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:21.455460 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.456190 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.457398 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.457612 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.457689 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.458650 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:21.458652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:21.458655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:21.458665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:21.458682 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:21.458689 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:21.458744 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.458871 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.458916 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:48:21.459816 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.451753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38815 (local address 127.2.182.3:58025)
0924 10:48:21.452361 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.452400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.453850 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.454451 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.454458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.454571 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:21.455296 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.455317 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.456316 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.456319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.457301 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.457310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.457734 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.457736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.457872 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.458425 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:21.458427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:21.458464 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:21.458522 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:21.458758 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:21.458764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:21.458768 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.458856 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.459068 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260924 10:48:21.460825 20802 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38815
I20260924 10:48:21.461071 20802 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:21.461555 20802 heartbeater.cc:511] Master 127.2.182.62:38815 requested a full tablet report, sending...
I20260924 10:48:21.462488 20352 ts_manager.cc:194] Registered new tserver with Master: 33f47636c3ec4b2e93f3aae68ebf384c (127.2.182.3:37989)
I20260924 10:48:21.463045 20352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:58025
I20260924 10:48:21.467646 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:21.472725 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.469520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.2.182.62:38815)
0924 10:48:21.469628 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.469632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.469773 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:48:21.469864 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.469866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.469900 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.469971 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:21.470207 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.470684 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.471441 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.471634 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.471705 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.472454 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:21.472457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:21.472458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:21.472467 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:21.472482 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:21.472488 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:21.472537 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.472590 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.472631 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:21.475378 20352 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:21.478360 20352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:21.492002 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.486965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37989 (local address 127.0.0.1:56476)
0924 10:48:21.487790 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.487812 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.487900 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.488306 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.488309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.488326 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:21.488405 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.488412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.489602 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.489622 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.490694 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.490701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.491447 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.491451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.491562 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.491598 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.491717 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.491828 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260924 10:48:21.492197 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.486585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.2.182.2:35343)
0924 10:48:21.487063 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.487075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.487098 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:48:21.487174 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.487179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.487388 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.487557 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:21.488048 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.489117 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.490632 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.491422 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.491515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.491683 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.491935 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.492036 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":139,"threads_started":1}
I20260924 10:48:21.492496 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.486248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:35343 (local address 127.0.0.1:34250)
0924 10:48:21.486580 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.486630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.486764 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.487734 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.487740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.487765 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:21.487910 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.487917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.489222 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.489227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.490507 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.490517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.491721 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.491725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.491802 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.491820 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.491892 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.491948 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260924 10:48:21.492826 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.486279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:33691 (local address 127.0.0.1:45136)
0924 10:48:21.487210 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.487224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.487312 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.488710 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.488713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.488729 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:21.488804 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.488809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.490115 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.490118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.491300 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.491311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.492380 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.492383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.492525 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.492550 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.492629 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.492709 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":97,"threads_started":1}
I20260924 10:48:21.492931 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.487130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.2.182.3:37989)
0924 10:48:21.487743 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.487754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.487905 (+ 151us) server_negotiation.cc:407] Connection header received
0924 10:48:21.488005 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.488013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.488178 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.488324 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:21.488496 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.489487 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.490802 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.491320 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.491457 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.491645 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.492638 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.492758 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":131,"threads_started":1}
I20260924 10:48:21.493920 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.487487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.2.182.1:33691)
0924 10:48:21.488153 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.488164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.488185 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:48:21.488254 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.488259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.488425 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.488571 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:21.488958 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.489993 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.491469 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.492201 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.493113 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.493645 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.493693 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.493775 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":130,"threads_started":1}
I20260924 10:48:21.495460 20605 tablet_service.cc:1467] Processing CreateTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 (DEFAULT_TABLE table=test-table [id=4463966dde8840ff88e932f4d9e5577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:21.495512 20736 tablet_service.cc:1467] Processing CreateTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 (DEFAULT_TABLE table=test-table [id=4463966dde8840ff88e932f4d9e5577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:21.496295 20605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa93d351f0ad47ceaf9d974580023fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:21.496428 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa93d351f0ad47ceaf9d974580023fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:21.496248 20474 tablet_service.cc:1467] Processing CreateTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 (DEFAULT_TABLE table=test-table [id=4463966dde8840ff88e932f4d9e5577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:21.497251 20474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa93d351f0ad47ceaf9d974580023fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:21.502476 20819 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Bootstrap starting.
I20260924 10:48:21.503638 20821 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Bootstrap starting.
I20260924 10:48:21.504151 20820 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Bootstrap starting.
I20260924 10:48:21.504704 20819 tablet_bootstrap.cc:654] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:21.505564 20819 log.cc:824] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:21.505646 20821 tablet_bootstrap.cc:654] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:21.505859 20820 tablet_bootstrap.cc:654] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:21.506304 20821 log.cc:824] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:21.506603 20820 log.cc:824] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:21.507776 20819 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: No bootstrap required, opened a new log
I20260924 10:48:21.507956 20819 ts_tablet_manager.cc:1397] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:48:21.508198 20821 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: No bootstrap required, opened a new log
I20260924 10:48:21.508383 20821 ts_tablet_manager.cc:1397] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:48:21.508420 20820 tablet_bootstrap.cc:492] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: No bootstrap required, opened a new log
I20260924 10:48:21.508572 20820 ts_tablet_manager.cc:1397] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:48:21.511412 20821 raft_consensus.cc:348] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.511649 20821 raft_consensus.cc:374] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:21.511700 20821 raft_consensus.cc:729] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7476ccc0c42440868647f70812a44204, State: Initialized, Role: FOLLOWER
I20260924 10:48:21.512133 20821 consensus_queue.cc:260] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.512148 20819 raft_consensus.cc:348] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.512372 20819 raft_consensus.cc:374] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:21.512436 20819 raft_consensus.cc:729] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e929e643e4d84f678b0c655efaf31474, State: Initialized, Role: FOLLOWER
I20260924 10:48:21.512485 20820 raft_consensus.cc:348] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.512759 20820 raft_consensus.cc:374] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:21.512805 20820 raft_consensus.cc:729] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f47636c3ec4b2e93f3aae68ebf384c, State: Initialized, Role: FOLLOWER
I20260924 10:48:21.512804 20540 heartbeater.cc:503] Master 127.2.182.62:38815 was elected leader, sending a full tablet report...
I20260924 10:48:21.512856 20819 consensus_queue.cc:260] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.513191 20820 consensus_queue.cc:260] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.513550 20671 heartbeater.cc:503] Master 127.2.182.62:38815 was elected leader, sending a full tablet report...
I20260924 10:48:21.513739 20819 ts_tablet_manager.cc:1428] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:48:21.513953 20821 ts_tablet_manager.cc:1428] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:48:21.513971 20820 ts_tablet_manager.cc:1428] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:48:21.515441 20802 heartbeater.cc:503] Master 127.2.182.62:38815 was elected leader, sending a full tablet report...
W20260924 10:48:21.564651 20541 tablet.cc:2369] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:21.608278 20826 raft_consensus.cc:482] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:21.608445 20826 raft_consensus.cc:504] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.609661 20826 leader_election.cc:288] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7476ccc0c42440868647f70812a44204 (127.2.182.1:33691), 33f47636c3ec4b2e93f3aae68ebf384c (127.2.182.3:37989)
I20260924 10:48:21.613358 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.609860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:33691 (local address 127.2.182.2:46301)
0924 10:48:21.610010 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.610042 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.610140 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.610609 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.610612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.610650 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:21.610766 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.610771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.611495 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.611501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.612280 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.612288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.612939 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.612943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.613032 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.613057 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.613159 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.613228 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1,"spinlock_wait_cycles":21248}
I20260924 10:48:21.613356 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.609986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:46301 (local address 127.2.182.1:33691)
0924 10:48:21.610123 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.610128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.610144 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:48:21.610447 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.610451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.610510 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.610597 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:21.610877 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.611384 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.612396 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.612814 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.612964 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.613027 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.613175 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.613239 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:48:21.613709 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.610304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:38667 (local address 127.2.182.3:37989)
0924 10:48:21.610436 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.610440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.610605 (+ 165us) server_negotiation.cc:407] Connection header received
0924 10:48:21.610775 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.610778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.610827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.610908 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:21.611158 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.611642 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.612653 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.613134 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.613245 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.613297 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.613529 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.613603 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:48:21.614005 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:37989 (local address 127.2.182.2:38667)
0924 10:48:21.610514 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.610526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:21.610671 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:21.610918 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:21.610920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:21.610935 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:21.611051 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.611057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.611751 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.611754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.612536 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:21.612544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.613254 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.613257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:21.613350 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.613376 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:48:21.613839 (+ 463us) client_negotiation.cc:240] Negotiation successful
0924 10:48:21.613909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"spinlock_wait_cycles":896,"thread_start_us":135,"threads_started":1}
I20260924 10:48:21.613981 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa93d351f0ad47ceaf9d974580023fd8" candidate_uuid: "e929e643e4d84f678b0c655efaf31474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7476ccc0c42440868647f70812a44204" is_pre_election: true
I20260924 10:48:21.614274 20494 raft_consensus.cc:2454] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e929e643e4d84f678b0c655efaf31474 in term 0.
I20260924 10:48:21.614678 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa93d351f0ad47ceaf9d974580023fd8" candidate_uuid: "e929e643e4d84f678b0c655efaf31474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" is_pre_election: true
I20260924 10:48:21.614758 20559 leader_election.cc:302] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7476ccc0c42440868647f70812a44204, e929e643e4d84f678b0c655efaf31474; no voters:
I20260924 10:48:21.614951 20756 raft_consensus.cc:2454] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e929e643e4d84f678b0c655efaf31474 in term 0.
I20260924 10:48:21.615012 20826 raft_consensus.cc:2781] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:21.615125 20826 raft_consensus.cc:482] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:21.615175 20826 raft_consensus.cc:3037] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:21.616204 20826 raft_consensus.cc:504] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.616536 20826 leader_election.cc:288] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [CANDIDATE]: Term 1 election: Requested vote from peers 7476ccc0c42440868647f70812a44204 (127.2.182.1:33691), 33f47636c3ec4b2e93f3aae68ebf384c (127.2.182.3:37989)
I20260924 10:48:21.616858 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa93d351f0ad47ceaf9d974580023fd8" candidate_uuid: "e929e643e4d84f678b0c655efaf31474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7476ccc0c42440868647f70812a44204"
I20260924 10:48:21.616979 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa93d351f0ad47ceaf9d974580023fd8" candidate_uuid: "e929e643e4d84f678b0c655efaf31474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f47636c3ec4b2e93f3aae68ebf384c"
I20260924 10:48:21.617003 20494 raft_consensus.cc:3037] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:21.617100 20756 raft_consensus.cc:3037] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:21.618000 20494 raft_consensus.cc:2454] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e929e643e4d84f678b0c655efaf31474 in term 1.
I20260924 10:48:21.618139 20756 raft_consensus.cc:2454] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e929e643e4d84f678b0c655efaf31474 in term 1.
I20260924 10:48:21.618317 20559 leader_election.cc:302] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7476ccc0c42440868647f70812a44204, e929e643e4d84f678b0c655efaf31474; no voters:
I20260924 10:48:21.618489 20826 raft_consensus.cc:2781] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:21.618739 20826 raft_consensus.cc:686] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 1 LEADER]: Becoming Leader. State: Replica: e929e643e4d84f678b0c655efaf31474, State: Running, Role: LEADER
I20260924 10:48:21.619071 20826 consensus_queue.cc:237] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } }
I20260924 10:48:21.622397 20350 catalog_manager.cc:5495] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 reported cstate change: term changed from 0 to 1, leader changed from <none> to e929e643e4d84f678b0c655efaf31474 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "e929e643e4d84f678b0c655efaf31474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e929e643e4d84f678b0c655efaf31474" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260924 10:48:21.643016 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.639021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.2.182.2:35343)
0924 10:48:21.639150 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:21.639154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:21.639199 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:48:21.639288 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:21.639290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:21.639385 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:21.639507 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:21.639747 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.640308 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.641439 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:21.641835 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:21.641950 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:21.642255 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:21.642662 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:21.642751 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:21.642877 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 10:48:21.642918 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:48:21.649351 20494 raft_consensus.cc:1261] T aa93d351f0ad47ceaf9d974580023fd8 P 7476ccc0c42440868647f70812a44204 [term 1 FOLLOWER]: Refusing update from remote peer e929e643e4d84f678b0c655efaf31474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:21.649351 20756 raft_consensus.cc:1261] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 1 FOLLOWER]: Refusing update from remote peer e929e643e4d84f678b0c655efaf31474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:21.650133 20826 consensus_queue.cc:1035] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7476ccc0c42440868647f70812a44204" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:21.650646 20829 consensus_queue.cc:1035] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f47636c3ec4b2e93f3aae68ebf384c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:21.658056 20832 mvcc.cc:205] Tried to move back new op lower bound from 7332851309148368896 to 7332851309034463232. Current Snapshot: MvccSnapshot[applied={T|T < 7332851309148368896}]
I20260924 10:48:21.658447 20833 mvcc.cc:205] Tried to move back new op lower bound from 7332851309148368896 to 7332851309034463232. Current Snapshot: MvccSnapshot[applied={T|T < 7332851309148368896}]
I20260924 10:48:21.658972 20834 mvcc.cc:205] Tried to move back new op lower bound from 7332851309148368896 to 7332851309034463232. Current Snapshot: MvccSnapshot[applied={T|T < 7332851309148368896}]
I20260924 10:48:21.667261 20352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:21.667414 20352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:21.669842 20352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16f9f0be719044bf90923d7d0bf7951d: Sending DeleteTablet for 3 replicas of tablet aa93d351f0ad47ceaf9d974580023fd8
I20260924 10:48:21.670565 20605 tablet_service.cc:1514] Processing DeleteTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:21 UTC) from {username='slave'} at 127.0.0.1:34250
I20260924 10:48:21.670735 20736 tablet_service.cc:1514] Processing DeleteTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:21 UTC) from {username='slave'} at 127.0.0.1:56476
I20260924 10:48:21.670737 20474 tablet_service.cc:1514] Processing DeleteTablet for tablet aa93d351f0ad47ceaf9d974580023fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:21 UTC) from {username='slave'} at 127.0.0.1:45136
I20260924 10:48:21.671047 20842 tablet_replica.cc:330] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: stopping tablet replica
I20260924 10:48:21.671329 20842 raft_consensus.cc:2229] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:21.671738 20842 raft_consensus.cc:2258] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:21.671985 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20412
I20260924 10:48:21.673369 20842 ts_tablet_manager.cc:1905] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:21.675498 20843 tablet_replica.cc:330] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: stopping tablet replica
I20260924 10:48:21.675781 20843 raft_consensus.cc:2229] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:21.676059 20843 raft_consensus.cc:2258] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:21.676407 20842 ts_tablet_manager.cc:1918] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:21.676519 20842 log.cc:1196] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/aa93d351f0ad47ceaf9d974580023fd8
I20260924 10:48:21.676801 20842 ts_tablet_manager.cc:1939] T aa93d351f0ad47ceaf9d974580023fd8 P e929e643e4d84f678b0c655efaf31474: Deleting consensus metadata
I20260924 10:48:21.677560 20843 ts_tablet_manager.cc:1905] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:21.677843 20339 catalog_manager.cc:4841] TS e929e643e4d84f678b0c655efaf31474 (127.2.182.2:35343): tablet aa93d351f0ad47ceaf9d974580023fd8 (table test-table [id=4463966dde8840ff88e932f4d9e5577e]) successfully deleted
I20260924 10:48:21.680085 20843 ts_tablet_manager.cc:1918] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:21.680174 20843 log.cc:1196] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/aa93d351f0ad47ceaf9d974580023fd8
I20260924 10:48:21.680502 20843 ts_tablet_manager.cc:1939] T aa93d351f0ad47ceaf9d974580023fd8 P 33f47636c3ec4b2e93f3aae68ebf384c: Deleting consensus metadata
I20260924 10:48:21.682065 20337 catalog_manager.cc:4841] TS 33f47636c3ec4b2e93f3aae68ebf384c (127.2.182.3:37989): tablet aa93d351f0ad47ceaf9d974580023fd8 (table test-table [id=4463966dde8840ff88e932f4d9e5577e]) successfully deleted
W20260924 10:48:21.682487 20337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:33691 (error 108)
I20260924 10:48:21.682845 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20543
I20260924 10:48:21.685299 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:21.685036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:33691 (local address 127.0.0.1:45142)
0924 10:48:21.685163 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:21.685228 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:48:21.685451 20337 catalog_manager.cc:4584] TS 7476ccc0c42440868647f70812a44204 (127.2.182.1:33691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa93d351f0ad47ceaf9d974580023fd8: Network error: Client connection negotiation failed: client connection to 127.2.182.1:33691: connect: Connection refused (error 111)
I20260924 10:48:21.689078 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20674
I20260924 10:48:21.694341 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20321
2026-09-24T10:48:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:21 dist-test-slave-6lwt krb5kdc[20848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:21 dist-test-slave-6lwt krb5kdc[20848](info): set up 2 sockets
Sep 24 10:48:21 dist-test-slave-6lwt krb5kdc[20848](info): commencing operation
krb5kdc: starting...
W20260924 10:48:22.701090 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:22 dist-test-slave-6lwt krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:22.818164 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36529
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:22.926038 20861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:22.926327 20861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:22.926401 20861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:22.930927 20861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:22.930982 20861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:22.931001 20861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:22.931018 20861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:22.931033 20861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:22.936168 20861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:22.937256 20861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:22.938266 20861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:22.946259 20866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:22.946408 20867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:22.946456 20869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:22.946563 20861 server_base.cc:1034] running on GCE node
I20260924 10:48:22.947101 20861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:22.948098 20861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:22.949275 20861 hybrid_clock.cc:648] HybridClock initialized: now 1790246902949267 us; error 33 us; skew 500 ppm
Sep 24 10:48:22 dist-test-slave-6lwt krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246902, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:22.951941 20861 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:22.952744 20861 webserver.cc:460] Webserver started at http://127.2.182.62:44815/ using document root <none> and password file <none>
I20260924 10:48:22.953310 20861 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:22.953370 20861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:22.953527 20861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:22.955401 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "5bcf77c19e9c4695aa2566d7d09bc733"
format_stamp: "Formatted at 2026-09-24 10:48:22 on dist-test-slave-6lwt"
server_key: "a516711fbce0e301b31906598321abe8"
server_key_iv: "a8e6bf547a4c191bbd66a0cb9c53ebe9"
server_key_version: "encryptionkey@0"
I20260924 10:48:22.955878 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "5bcf77c19e9c4695aa2566d7d09bc733"
format_stamp: "Formatted at 2026-09-24 10:48:22 on dist-test-slave-6lwt"
server_key: "a516711fbce0e301b31906598321abe8"
server_key_iv: "a8e6bf547a4c191bbd66a0cb9c53ebe9"
server_key_version: "encryptionkey@0"
I20260924 10:48:22.959349 20861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:22.961604 20876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:22.962759 20861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:22.962858 20861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "5bcf77c19e9c4695aa2566d7d09bc733"
format_stamp: "Formatted at 2026-09-24 10:48:22 on dist-test-slave-6lwt"
server_key: "a516711fbce0e301b31906598321abe8"
server_key_iv: "a8e6bf547a4c191bbd66a0cb9c53ebe9"
server_key_version: "encryptionkey@0"
I20260924 10:48:22.962939 20861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:22.985674 20861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:22.988449 20861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:22.988607 20861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:22.995312 20861 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36529
I20260924 10:48:22.995337 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36529 every 8 connection(s)
I20260924 10:48:22.996201 20861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:22.998826 20929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:23.003854 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20861
I20260924 10:48:23.003991 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:23.004165 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: Bootstrap starting.
I20260924 10:48:23.004287 2776 external_mini_cluster.cc:1428] Setting key 8f3c5b3596cac92b99332c73a90b81c2
I20260924 10:48:23.006650 20929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:23.007386 20929 log.cc:824] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:23.009253 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: No bootstrap required, opened a new log
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:23.012751 20929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } }
I20260924 10:48:23.013021 20929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:23.013105 20929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcf77c19e9c4695aa2566d7d09bc733, State: Initialized, Role: FOLLOWER
I20260924 10:48:23.013597 20929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } }
I20260924 10:48:23.013762 20929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:23.013856 20929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:23.013952 20929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:23.014911 20929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } }
I20260924 10:48:23.015244 20929 leader_election.cc:302] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bcf77c19e9c4695aa2566d7d09bc733; no voters:
I20260924 10:48:23.015524 20929 leader_election.cc:288] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:23.015576 20934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:23.015780 20934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [term 1 LEADER]: Becoming Leader. State: Replica: 5bcf77c19e9c4695aa2566d7d09bc733, State: Running, Role: LEADER
I20260924 10:48:23.016054 20934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } }
I20260924 10:48:23.016726 20929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:23.017812 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } } }
I20260924 10:48:23.017993 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:23.018358 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.005628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.2.182.62:36529)
0924 10:48:23.006027 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.006041 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.006074 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:48:23.006660 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.006675 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.007076 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.007390 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:23.008203 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.009233 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.010049 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.010517 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.010683 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.013131 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:23.013139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:23.013156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:23.013207 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:23.015091 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.015566 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.015572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.015578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.015715 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.016064 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.016069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.016090 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.016530 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:23.016728 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.017017 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.017329 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260924 10:48:23.019459 20936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bcf77c19e9c4695aa2566d7d09bc733. Latest consensus state: current_term: 1 leader_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcf77c19e9c4695aa2566d7d09bc733" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36529 } } }
I20260924 10:48:23.020323 20936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733 [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:23.021809 20950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:23.021867 20950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:23.021938 20943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:23.022640 20943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:23.027499 20943 catalog_manager.cc:1347] Generated new cluster ID: 1a8dff837d034ce88ed3385350987909
I20260924 10:48:23.027573 20943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:23.035197 20943 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:23.036051 20943 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:23.045928 20943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: Generated new TSK 0
I20260924 10:48:23.046578 20943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:48:23.073930 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:23.187088 20955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:23.187335 20955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:23.187376 20955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:23.191414 20955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:23.191468 20955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:23.191571 20955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:23.196486 20955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:23.197535 20955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:23.198526 20955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:23.207784 20955 server_base.cc:1034] running on GCE node
W20260924 10:48:23.207743 20960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:23.207739 20961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:23.207739 20963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:23.208407 20955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:23.209136 20955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:23.210292 20955 hybrid_clock.cc:648] HybridClock initialized: now 1790246903210280 us; error 23 us; skew 500 ppm
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:23.213027 20955 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:48:23.213811 20955 webserver.cc:460] Webserver started at http://127.2.182.1:38383/ using document root <none> and password file <none>
I20260924 10:48:23.214375 20955 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:23.214466 20955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:23.214675 20955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:23.216585 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "9dc2058775b94f2e92178dc405fa8fc7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "231787743bda1e191474b63642b1f82a"
server_key_iv: "b18f00d3944a779d0a8221218e82c52e"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.217236 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "9dc2058775b94f2e92178dc405fa8fc7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "231787743bda1e191474b63642b1f82a"
server_key_iv: "b18f00d3944a779d0a8221218e82c52e"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.220809 20955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:23.223105 20970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.224193 20955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:23.224332 20955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "9dc2058775b94f2e92178dc405fa8fc7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "231787743bda1e191474b63642b1f82a"
server_key_iv: "b18f00d3944a779d0a8221218e82c52e"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.224444 20955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:23.237860 20955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:23.241339 20955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:23.241541 20955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:23.242115 20955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:23.242911 20955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:23.242983 20955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.243064 20955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:23.243126 20955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.251679 20955 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:41943
I20260924 10:48:23.251698 21083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:41943 every 8 connection(s)
I20260924 10:48:23.252557 20955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:23.259831 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 20955
I20260924 10:48:23.259971 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:23.260281 2776 external_mini_cluster.cc:1428] Setting key 093dad5e11f034333e5e9c1c689bd200
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:23.265699 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.254376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:60765 (local address 127.2.182.62:36529)
0924 10:48:23.254540 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.254545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.255363 (+ 818us) server_negotiation.cc:407] Connection header received
0924 10:48:23.256177 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.256180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.256224 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.256298 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:23.257458 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.257987 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.259025 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.259253 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.259320 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.262020 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:23.262023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:23.262026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:23.262049 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:23.263608 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.264095 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.264102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.264105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.264201 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.264518 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.264523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.264528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.264745 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:23.264857 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.265358 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.265488 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:48:23.266450 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.254759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36529 (local address 127.2.182.1:60765)
0924 10:48:23.255207 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.255249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.255976 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.256458 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.256465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.256798 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:23.257334 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.257347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.258088 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.258091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.258916 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.258924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.259366 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.259368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.259496 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.260197 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:23.260212 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:23.261859 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:23.263728 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.263732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.263742 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.263996 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.264298 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.264300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.264301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.264424 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.264854 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:23.264861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:23.265131 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.265319 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.265639 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260924 10:48:23.267714 21084 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36529
I20260924 10:48:23.267947 21084 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:23.268411 21084 heartbeater.cc:511] Master 127.2.182.62:36529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:48:23.269878 20893 ts_manager.cc:194] Registered new tserver with Master: 9dc2058775b94f2e92178dc405fa8fc7 (127.2.182.1:41943)
I20260924 10:48:23.271024 20893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:60765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:48:23.286312 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:23.395366 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:23.395619 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:23.395661 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:23.399492 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:23.399542 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:23.399596 21089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:23.404999 21089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:23.405948 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:23.406862 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:23.416270 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:23.416426 21089 server_base.cc:1034] running on GCE node
W20260924 10:48:23.416260 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:23.416332 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:23.417086 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:23.417768 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:23.418931 21089 hybrid_clock.cc:648] HybridClock initialized: now 1790246903418908 us; error 33 us; skew 500 ppm
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:23.425207 21089 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:48:23.426100 21089 webserver.cc:460] Webserver started at http://127.2.182.2:34837/ using document root <none> and password file <none>
I20260924 10:48:23.426681 21089 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:23.426743 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:23.426985 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:23.429160 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "371d566d408d4d5c894b9cf93bb783f3"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "11f87e133b1745e126ff6db08bbb26ba"
server_key_iv: "b429a56445d63a8204afce00fefd1ade"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.429723 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "371d566d408d4d5c894b9cf93bb783f3"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "11f87e133b1745e126ff6db08bbb26ba"
server_key_iv: "b429a56445d63a8204afce00fefd1ade"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.433418 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:48:23.435717 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.436961 21089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:23.437104 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "371d566d408d4d5c894b9cf93bb783f3"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "11f87e133b1745e126ff6db08bbb26ba"
server_key_iv: "b429a56445d63a8204afce00fefd1ade"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.437212 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:23.451280 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:23.454371 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:23.454586 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:23.455153 21089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:23.455942 21089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:23.456012 21089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.456082 21089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:23.456125 21089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.465103 21089 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:44295
I20260924 10:48:23.465133 21217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:44295 every 8 connection(s)
I20260924 10:48:23.465948 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:23.472393 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21089
I20260924 10:48:23.472541 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:23.472810 2776 external_mini_cluster.cc:1428] Setting key 3bd25439113d6fcb0cd5479aa1910c90
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:23.478931 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.467601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:50817 (local address 127.2.182.62:36529)
0924 10:48:23.467739 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.467743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.468561 (+ 818us) server_negotiation.cc:407] Connection header received
0924 10:48:23.469514 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.469517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.469571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.469678 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:23.470791 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.471320 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.472472 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.472787 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.472925 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.475477 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:23.475480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:23.475483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:23.475517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:23.476857 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.477452 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.477455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.477457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.477540 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.477877 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.477880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.477882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.478061 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:23.478168 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.478635 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.478750 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:23.479739 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.467966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36529 (local address 127.2.182.2:50817)
0924 10:48:23.468420 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.468461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.469306 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.469849 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.469856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.470193 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:23.470650 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.470662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.471456 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.471460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.472307 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.472322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.472958 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.472963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.473133 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.473762 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:23.473781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:23.475299 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:23.477060 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.477065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.477079 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.477325 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.477631 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.477635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.477637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.477774 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.478166 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:23.478172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:23.478412 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.478594 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.478896 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260924 10:48:23.480813 21218 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36529
I20260924 10:48:23.481081 21218 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:23.481650 21218 heartbeater.cc:511] Master 127.2.182.62:36529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:48:23.482733 20893 ts_manager.cc:194] Registered new tserver with Master: 371d566d408d4d5c894b9cf93bb783f3 (127.2.182.2:44295)
I20260924 10:48:23.483404 20893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:50817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:48:23.499159 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:23.605890 21223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:23.606125 21223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:23.606165 21223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:23.609962 21223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:23.610010 21223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:23.610080 21223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:23.615298 21223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:23.616240 21223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:23.617274 21223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:23.626515 21228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:23.626503 21231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:23.626499 21229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:23.626633 21223 server_base.cc:1034] running on GCE node
I20260924 10:48:23.627226 21223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:23.627837 21223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:23.628988 21223 hybrid_clock.cc:648] HybridClock initialized: now 1790246903628963 us; error 25 us; skew 500 ppm
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:23.631547 21223 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:48:23.632390 21223 webserver.cc:460] Webserver started at http://127.2.182.3:33391/ using document root <none> and password file <none>
I20260924 10:48:23.632972 21223 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:23.633040 21223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:23.633255 21223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:23.635584 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "617fe94f17b5431395181977e25d7ac7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "cf9f3054d81faa97b202dd9d9c12abeb"
server_key_iv: "f7913c95968c2b64f57a65cab4d4d602"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.636194 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "617fe94f17b5431395181977e25d7ac7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "cf9f3054d81faa97b202dd9d9c12abeb"
server_key_iv: "f7913c95968c2b64f57a65cab4d4d602"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.639743 21223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:23.642081 21238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.643224 21223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:23.643342 21223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "617fe94f17b5431395181977e25d7ac7"
format_stamp: "Formatted at 2026-09-24 10:48:23 on dist-test-slave-6lwt"
server_key: "cf9f3054d81faa97b202dd9d9c12abeb"
server_key_iv: "f7913c95968c2b64f57a65cab4d4d602"
server_key_version: "encryptionkey@0"
I20260924 10:48:23.643437 21223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:23.662653 21223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:23.665478 21223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:23.665666 21223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:23.666265 21223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:23.667163 21223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:23.667208 21223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.667248 21223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:23.667299 21223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:23.676079 21223 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:45531
I20260924 10:48:23.676141 21351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:45531 every 8 connection(s)
I20260924 10:48:23.677038 21223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:48:23.684842 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21223
I20260924 10:48:23.685005 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:23.685326 2776 external_mini_cluster.cc:1428] Setting key e5b51a7ef23580bd9828f7b7b63881c1
Sep 24 10:48:23 dist-test-slave-6lwt krb5kdc[20848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:23.690546 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.678778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:53735 (local address 127.2.182.62:36529)
0924 10:48:23.678918 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.678922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.679706 (+ 784us) server_negotiation.cc:407] Connection header received
0924 10:48:23.680589 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.680592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.680648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.680758 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:23.681923 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.682455 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.683582 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.683803 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.683914 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.686720 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:23.686724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:23.686726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:23.686754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:23.688403 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.689000 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.689004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.689005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.689089 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.689437 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.689440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.689442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.689672 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:23.689767 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.690258 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.690375 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:48:23.691287 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.679145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36529 (local address 127.2.182.3:53735)
0924 10:48:23.679569 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.679609 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.680387 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.680983 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.680990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.681329 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:23.681784 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.681795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.682578 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.682588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.683459 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.683467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.683956 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.683960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.684125 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.684742 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:23.684757 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:23.686569 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:23.688563 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.688568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.688584 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.688881 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.689190 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:23.689193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:23.689195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:23.689327 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:23.689793 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:23.689799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:23.690025 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.690203 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.690515 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260924 10:48:23.692374 21352 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36529
I20260924 10:48:23.692575 21352 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:23.693084 21352 heartbeater.cc:511] Master 127.2.182.62:36529 requested a full tablet report, sending...
I20260924 10:48:23.693981 20893 ts_manager.cc:194] Registered new tserver with Master: 617fe94f17b5431395181977e25d7ac7 (127.2.182.3:45531)
I20260924 10:48:23.694489 20893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:53735
I20260924 10:48:23.698987 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:23.707610 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.700869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.2.182.62:36529)
0924 10:48:23.701003 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.701007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.701188 (+ 181us) server_negotiation.cc:407] Connection header received
0924 10:48:23.701281 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.701284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.701317 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.701380 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:23.702008 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.702472 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.703238 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.703435 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.703506 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.704980 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:23.704983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:23.704985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:23.705005 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:23.706178 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.706620 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.706623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.706624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.706689 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:23.706939 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:23.706941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:23.706942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:23.707107 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:23.707211 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.707414 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.707481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:48:23.710302 20893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:23.713253 20893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:23.727443 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.721275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:44295 (local address 127.0.0.1:60214)
0924 10:48:23.721593 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.721634 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.721760 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.722661 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.722668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.722696 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:23.722852 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.722860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.724433 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.724436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.725768 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.725777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.726846 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.726852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.726987 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.727036 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.727140 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.727266 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":30,"thread_start_us":95,"threads_started":1}
I20260924 10:48:23.727475 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.721416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.2.182.2:44295)
0924 10:48:23.722113 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.722129 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.722155 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:48:23.722239 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.722247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.722488 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.723002 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:23.723183 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.724307 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.725893 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.726721 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.726833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.726984 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.727188 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.727293 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":142,"threads_started":1}
I20260924 10:48:23.727617 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.721540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:41943 (local address 127.0.0.1:50178)
0924 10:48:23.722053 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.722065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.722148 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.723169 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.723172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.723189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:23.723289 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.723297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.724245 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.724252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.725996 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.726004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.727201 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.727205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.727307 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.727328 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.727397 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.727463 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":155,"thread_start_us":88,"threads_started":1}
I20260924 10:48:23.727893 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.722573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.2.182.3:45531)
0924 10:48:23.722978 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.722990 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.723009 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:48:23.723111 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.723116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.723289 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.723672 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:23.723809 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.724775 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.726305 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.726891 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.727257 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.727393 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.727441 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.727742 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260924 10:48:23.728112 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.722074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.2.182.1:41943)
0924 10:48:23.722769 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.722780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.722802 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:48:23.722870 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.722882 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.723027 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.723195 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:23.723411 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.724120 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.726129 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.726887 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.726997 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.727138 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.727831 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.727930 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":125,"threads_started":1}
I20260924 10:48:23.728788 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.721969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:45531 (local address 127.0.0.1:36926)
0924 10:48:23.722351 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.722363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.722710 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.723438 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.723441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.723459 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:23.723536 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.723541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.725140 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.725146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.726201 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.726207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.727020 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.727024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.727108 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.727140 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.727181 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.728657 (+ 1476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":1440,"thread_start_us":79,"threads_started":1}
I20260924 10:48:23.730736 21152 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac8d3ef47244c0082f99745bf04739b (DEFAULT_TABLE table=test-table [id=72a8fd311ea845869943d30da4d1a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:23.730901 21018 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac8d3ef47244c0082f99745bf04739b (DEFAULT_TABLE table=test-table [id=72a8fd311ea845869943d30da4d1a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:23.731050 21286 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac8d3ef47244c0082f99745bf04739b (DEFAULT_TABLE table=test-table [id=72a8fd311ea845869943d30da4d1a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:23.731729 21152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac8d3ef47244c0082f99745bf04739b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:23.731909 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac8d3ef47244c0082f99745bf04739b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:23.731909 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac8d3ef47244c0082f99745bf04739b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:23.737553 21369 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Bootstrap starting.
I20260924 10:48:23.739087 21370 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Bootstrap starting.
I20260924 10:48:23.739748 21371 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Bootstrap starting.
I20260924 10:48:23.740732 21369 tablet_bootstrap.cc:654] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:23.741235 21370 tablet_bootstrap.cc:654] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:23.741706 21369 log.cc:824] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:23.742038 21371 tablet_bootstrap.cc:654] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:23.742347 21370 log.cc:824] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:23.742858 21371 log.cc:824] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:23.743872 21369 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: No bootstrap required, opened a new log
I20260924 10:48:23.744072 21369 ts_tablet_manager.cc:1397] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:48:23.744634 21370 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: No bootstrap required, opened a new log
I20260924 10:48:23.744678 21371 tablet_bootstrap.cc:492] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: No bootstrap required, opened a new log
I20260924 10:48:23.744809 21370 ts_tablet_manager.cc:1397] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:48:23.744874 21371 ts_tablet_manager.cc:1397] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:48:23.747742 21371 raft_consensus.cc:348] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.748025 21371 raft_consensus.cc:374] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:23.747848 21369 raft_consensus.cc:348] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.748085 21371 raft_consensus.cc:729] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617fe94f17b5431395181977e25d7ac7, State: Initialized, Role: FOLLOWER
I20260924 10:48:23.748090 21369 raft_consensus.cc:374] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:23.748147 21369 raft_consensus.cc:729] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371d566d408d4d5c894b9cf93bb783f3, State: Initialized, Role: FOLLOWER
I20260924 10:48:23.748376 21370 raft_consensus.cc:348] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.748586 21370 raft_consensus.cc:374] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:23.748503 21371 consensus_queue.cc:260] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.748546 21369 consensus_queue.cc:260] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.748636 21370 raft_consensus.cc:729] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc2058775b94f2e92178dc405fa8fc7, State: Initialized, Role: FOLLOWER
I20260924 10:48:23.749019 21370 consensus_queue.cc:260] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.749117 21352 heartbeater.cc:503] Master 127.2.182.62:36529 was elected leader, sending a full tablet report...
I20260924 10:48:23.749490 21371 ts_tablet_manager.cc:1428] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:48:23.749497 21369 ts_tablet_manager.cc:1428] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:48:23.749791 21370 ts_tablet_manager.cc:1428] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:23.749789 21218 heartbeater.cc:503] Master 127.2.182.62:36529 was elected leader, sending a full tablet report...
I20260924 10:48:23.750110 21084 heartbeater.cc:503] Master 127.2.182.62:36529 was elected leader, sending a full tablet report...
W20260924 10:48:23.753695 21085 tablet.cc:2369] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:23.830929 21376 raft_consensus.cc:482] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:23.831158 21376 raft_consensus.cc:504] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.832324 21376 leader_election.cc:288] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617fe94f17b5431395181977e25d7ac7 (127.2.182.3:45531), 9dc2058775b94f2e92178dc405fa8fc7 (127.2.182.1:41943)
I20260924 10:48:23.837363 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.832617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:40503 (local address 127.2.182.3:45531)
0924 10:48:23.832788 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.832792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.832807 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:48:23.833000 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.833003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.833049 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.833158 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:23.833456 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.834279 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836068 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.836675 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836784 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.836844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.837165 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.837236 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 10:48:23.837363 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.832728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:48607 (local address 127.2.182.1:41943)
0924 10:48:23.832848 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.832852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.833137 (+ 285us) server_negotiation.cc:407] Connection header received
0924 10:48:23.833245 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.833248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.833297 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.833354 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:23.833613 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.834316 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836068 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.836675 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836783 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.836844 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.837166 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.837236 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:48:23.837414 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.832533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:45531 (local address 127.2.182.2:40503)
0924 10:48:23.832731 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.832745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.832838 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.833186 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.833191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.833210 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:23.833324 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.833328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.834420 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.834423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.835943 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.835952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.836793 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.836957 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.837034 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.837117 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.837267 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":248}
I20260924 10:48:23.837608 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.832606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:41943 (local address 127.2.182.2:48607)
0924 10:48:23.833027 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.833037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:23.833154 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:23.833385 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:23.833388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:23.833409 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:23.833513 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.833518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.834426 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.834428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.835941 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:23.835952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.836793 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.836797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:23.837040 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.837059 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:48:23.837127 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:48:23.837267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":228,"thread_start_us":134,"threads_started":1}
I20260924 10:48:23.838106 21306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac8d3ef47244c0082f99745bf04739b" candidate_uuid: "371d566d408d4d5c894b9cf93bb783f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617fe94f17b5431395181977e25d7ac7" is_pre_election: true
I20260924 10:48:23.838229 21038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac8d3ef47244c0082f99745bf04739b" candidate_uuid: "371d566d408d4d5c894b9cf93bb783f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc2058775b94f2e92178dc405fa8fc7" is_pre_election: true
I20260924 10:48:23.838436 21306 raft_consensus.cc:2454] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371d566d408d4d5c894b9cf93bb783f3 in term 0.
I20260924 10:48:23.838550 21038 raft_consensus.cc:2454] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371d566d408d4d5c894b9cf93bb783f3 in term 0.
I20260924 10:48:23.838884 21106 leader_election.cc:302] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371d566d408d4d5c894b9cf93bb783f3, 617fe94f17b5431395181977e25d7ac7; no voters:
I20260924 10:48:23.839144 21376 raft_consensus.cc:2781] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:23.839223 21376 raft_consensus.cc:482] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:23.839265 21376 raft_consensus.cc:3037] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:23.840422 21376 raft_consensus.cc:504] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.840802 21376 leader_election.cc:288] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [CANDIDATE]: Term 1 election: Requested vote from peers 617fe94f17b5431395181977e25d7ac7 (127.2.182.3:45531), 9dc2058775b94f2e92178dc405fa8fc7 (127.2.182.1:41943)
I20260924 10:48:23.841109 21306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac8d3ef47244c0082f99745bf04739b" candidate_uuid: "371d566d408d4d5c894b9cf93bb783f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617fe94f17b5431395181977e25d7ac7"
I20260924 10:48:23.841164 21038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac8d3ef47244c0082f99745bf04739b" candidate_uuid: "371d566d408d4d5c894b9cf93bb783f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc2058775b94f2e92178dc405fa8fc7"
I20260924 10:48:23.841228 21306 raft_consensus.cc:3037] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:23.841298 21038 raft_consensus.cc:3037] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:23.842255 21038 raft_consensus.cc:2454] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371d566d408d4d5c894b9cf93bb783f3 in term 1.
I20260924 10:48:23.842269 21306 raft_consensus.cc:2454] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371d566d408d4d5c894b9cf93bb783f3 in term 1.
I20260924 10:48:23.842528 21106 leader_election.cc:302] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371d566d408d4d5c894b9cf93bb783f3, 617fe94f17b5431395181977e25d7ac7; no voters:
I20260924 10:48:23.842674 21376 raft_consensus.cc:2781] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:23.842885 21376 raft_consensus.cc:686] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 1 LEADER]: Becoming Leader. State: Replica: 371d566d408d4d5c894b9cf93bb783f3, State: Running, Role: LEADER
I20260924 10:48:23.843227 21376 consensus_queue.cc:237] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } }
I20260924 10:48:23.846186 20893 catalog_manager.cc:5495] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371d566d408d4d5c894b9cf93bb783f3 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "371d566d408d4d5c894b9cf93bb783f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371d566d408d4d5c894b9cf93bb783f3" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 44295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 } health_report { overall_health: UNKNOWN } } }
I20260924 10:48:23.878173 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.874986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.2.182.2:44295)
0924 10:48:23.875115 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:23.875119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:23.875132 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:48:23.875197 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:23.875199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:23.875247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:23.875351 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:23.875570 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.876054 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.876830 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:23.877039 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:23.877109 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:23.877371 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:23.877801 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:23.877932 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:23.878018 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:48:23.878054 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:23.883658 21306 raft_consensus.cc:1261] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 1 FOLLOWER]: Refusing update from remote peer 371d566d408d4d5c894b9cf93bb783f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:23.883659 21038 raft_consensus.cc:1261] T 6ac8d3ef47244c0082f99745bf04739b P 9dc2058775b94f2e92178dc405fa8fc7 [term 1 FOLLOWER]: Refusing update from remote peer 371d566d408d4d5c894b9cf93bb783f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:23.884418 21379 consensus_queue.cc:1035] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc2058775b94f2e92178dc405fa8fc7" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:23.884579 21376 consensus_queue.cc:1035] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "617fe94f17b5431395181977e25d7ac7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:23.890507 21382 mvcc.cc:205] Tried to move back new op lower bound from 7332851318300749824 to 7332851318144397312. Current Snapshot: MvccSnapshot[applied={T|T < 7332851318300749824}]
I20260924 10:48:23.891455 21383 mvcc.cc:205] Tried to move back new op lower bound from 7332851318300749824 to 7332851318144397312. Current Snapshot: MvccSnapshot[applied={T|T < 7332851318300749824}]
I20260924 10:48:23.898667 20892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:23.898829 20892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:23.901170 20892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bcf77c19e9c4695aa2566d7d09bc733: Sending DeleteTablet for 3 replicas of tablet 6ac8d3ef47244c0082f99745bf04739b
I20260924 10:48:23.901988 21018 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac8d3ef47244c0082f99745bf04739b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:50178
I20260924 10:48:23.902004 21286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac8d3ef47244c0082f99745bf04739b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:36926
I20260924 10:48:23.902109 21152 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac8d3ef47244c0082f99745bf04739b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:23 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:60214
I20260924 10:48:23.902522 21394 tablet_replica.cc:330] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: stopping tablet replica
I20260924 10:48:23.902523 21393 tablet_replica.cc:330] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: stopping tablet replica
I20260924 10:48:23.902786 21393 raft_consensus.cc:2229] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:23.902784 21394 raft_consensus.cc:2229] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:23.903008 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20955
I20260924 10:48:23.903087 21393 raft_consensus.cc:2258] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:23.903179 21394 raft_consensus.cc:2258] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:23.905123 21393 ts_tablet_manager.cc:1905] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:23.908288 21394 ts_tablet_manager.cc:1905] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:23.908435 21393 ts_tablet_manager.cc:1918] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:23.908531 21393 log.cc:1196] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/6ac8d3ef47244c0082f99745bf04739b
I20260924 10:48:23.908849 21393 ts_tablet_manager.cc:1939] T 6ac8d3ef47244c0082f99745bf04739b P 617fe94f17b5431395181977e25d7ac7: Deleting consensus metadata
I20260924 10:48:23.910173 20878 catalog_manager.cc:4841] TS 617fe94f17b5431395181977e25d7ac7 (127.2.182.3:45531): tablet 6ac8d3ef47244c0082f99745bf04739b (table test-table [id=72a8fd311ea845869943d30da4d1a473]) successfully deleted
W20260924 10:48:23.910728 20877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:41943 (error 108)
I20260924 10:48:23.911137 21394 ts_tablet_manager.cc:1918] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:23.911227 21394 log.cc:1196] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/6ac8d3ef47244c0082f99745bf04739b
I20260924 10:48:23.911540 21394 ts_tablet_manager.cc:1939] T 6ac8d3ef47244c0082f99745bf04739b P 371d566d408d4d5c894b9cf93bb783f3: Deleting consensus metadata
I20260924 10:48:23.911777 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.911482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:41943 (local address 127.0.0.1:50192)
0924 10:48:23.911609 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.911693 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:41943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:48:23.911767 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21089
W20260924 10:48:23.912488 20877 catalog_manager.cc:4584] TS 9dc2058775b94f2e92178dc405fa8fc7 (127.2.182.1:41943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ac8d3ef47244c0082f99745bf04739b: Network error: Client connection negotiation failed: client connection to 127.2.182.1:41943: connect: Connection refused (error 111)
I20260924 10:48:23.917723 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21223
I20260924 10:48:23.918897 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:23.917845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:44295 (local address 127.0.0.1:60236)
0924 10:48:23.918740 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:23.918803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":800}
I20260924 10:48:23.923318 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 20861
2026-09-24T10:48:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2229 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:23Z Disabled control of system clock
I20260924 10:48:23.952466 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38147
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:24.055071 21399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:24.055294 21399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:24.055328 21399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:24.059284 21399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:24.059330 21399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:24.059345 21399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:24.059379 21399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:24.059394 21399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:24.063925 21399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:24.064760 21399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:24.065971 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:24.073807 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.073861 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.073864 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.074160 21399 server_base.cc:1034] running on GCE node
I20260924 10:48:24.074712 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:24.075744 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:24.076974 21399 hybrid_clock.cc:648] HybridClock initialized: now 1790246904076956 us; error 28 us; skew 500 ppm
I20260924 10:48:24.078593 21399 webserver.cc:460] Webserver started at http://127.2.182.62:45583/ using document root <none> and password file <none>
I20260924 10:48:24.079139 21399 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:24.079236 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:24.079443 21399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:24.081430 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "6c1b123232a7b064dd230fef172e8224"
server_key_iv: "f1e00e1ea223004655bf884b7981369e"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.081970 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "6c1b123232a7b064dd230fef172e8224"
server_key_iv: "f1e00e1ea223004655bf884b7981369e"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.085332 21399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:24.087584 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.088713 21399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:24.088842 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "6c1b123232a7b064dd230fef172e8224"
server_key_iv: "f1e00e1ea223004655bf884b7981369e"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.089006 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:24.112501 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:24.113112 21399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:24.113312 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:24.119863 21399 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38147
I20260924 10:48:24.119877 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:38147 every 8 connection(s)
I20260924 10:48:24.120708 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:24.123574 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:24.127715 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21399
I20260924 10:48:24.127858 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:24.128125 2776 external_mini_cluster.cc:1428] Setting key 46313818188d9a4ef70925c53d04a80e
I20260924 10:48:24.128691 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: Bootstrap starting.
I20260924 10:48:24.130874 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:24.131734 21466 log.cc:824] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:24.133653 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: No bootstrap required, opened a new log
I20260924 10:48:24.136999 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.129414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.2.182.62:38147)
0924 10:48:24.129967 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.129980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.130013 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:48:24.130620 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.130635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.131052 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.131312 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:24.131703 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.132834 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.133977 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.134413 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.134531 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.135357 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:24.135362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:24.135373 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:24.135721 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:24.135771 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:24.135788 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:24.135859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.136013 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.136282 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":115,"threads_started":1}
I20260924 10:48:24.137507 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } }
I20260924 10:48:24.137697 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:24.137758 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fb6614bcaf43e3a88d1a8ca87032f1, State: Initialized, Role: FOLLOWER
I20260924 10:48:24.138141 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } }
I20260924 10:48:24.138280 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:24.138360 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:24.138473 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:24.139261 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } }
I20260924 10:48:24.139565 21466 leader_election.cc:302] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7fb6614bcaf43e3a88d1a8ca87032f1; no voters:
I20260924 10:48:24.139825 21466 leader_election.cc:288] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:24.139927 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:24.140154 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [term 1 LEADER]: Becoming Leader. State: Replica: a7fb6614bcaf43e3a88d1a8ca87032f1, State: Running, Role: LEADER
I20260924 10:48:24.140439 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } }
I20260924 10:48:24.140964 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:24.141957 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } } }
I20260924 10:48:24.142078 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:24.142402 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7fb6614bcaf43e3a88d1a8ca87032f1. Latest consensus state: current_term: 1 leader_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fb6614bcaf43e3a88d1a8ca87032f1" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38147 } } }
I20260924 10:48:24.142482 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:24.142501 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:24.145255 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:24.150178 21480 catalog_manager.cc:1347] Generated new cluster ID: 06742ea84e5d4370823c4a9eaa88ded3
I20260924 10:48:24.150229 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:24.170828 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:24.171949 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:24.178331 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: Generated new TSK 0
I20260924 10:48:24.179056 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:24.185539 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--builtin_ntp_servers=127.2.182.20:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:24.292770 21490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:24.293044 21490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:24.293082 21490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:24.296531 21490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:24.296567 21490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:24.296613 21490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:24.300968 21490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:24.301810 21490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:24.302958 21490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:24.311676 21496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.311734 21490 server_base.cc:1034] running on GCE node
W20260924 10:48:24.311708 21498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.311928 21495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.312289 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:24.312835 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:24.314013 21490 hybrid_clock.cc:648] HybridClock initialized: now 1790246904314000 us; error 23 us; skew 500 ppm
I20260924 10:48:24.315815 21490 webserver.cc:460] Webserver started at http://127.2.182.1:44531/ using document root <none> and password file <none>
I20260924 10:48:24.316335 21490 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:24.316431 21490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:24.316648 21490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:24.318555 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "d67140590beb44af99396eb6d99ad9b4"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2d8f24478386889cf8ee5eabb1115cf5"
server_key_iv: "570471f184ef046dee98a70d8156d43f"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.319083 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "d67140590beb44af99396eb6d99ad9b4"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2d8f24478386889cf8ee5eabb1115cf5"
server_key_iv: "570471f184ef046dee98a70d8156d43f"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.322729 21490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:24.325019 21504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.326100 21490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:24.326227 21490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "d67140590beb44af99396eb6d99ad9b4"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2d8f24478386889cf8ee5eabb1115cf5"
server_key_iv: "570471f184ef046dee98a70d8156d43f"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.326327 21490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:24.350494 21490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:24.351135 21490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:24.351316 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:24.351852 21490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:24.352725 21490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:24.352778 21490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.352834 21490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:24.352860 21490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.361187 21490 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:39625
I20260924 10:48:24.361224 21617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:39625 every 8 connection(s)
I20260924 10:48:24.361994 21490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:24.370769 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.363664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:34757 (local address 127.2.182.62:38147)
0924 10:48:24.363805 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.363810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.364818 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 10:48:24.366244 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.366247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.366304 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.366382 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:24.367199 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.367723 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.368795 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.369108 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.369216 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.370295 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:24.370298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:24.370301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:24.370313 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:24.370335 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:24.370348 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:24.370429 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.370579 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.370636 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:24.371275 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21490
I20260924 10:48:24.371387 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:24.371676 2776 external_mini_cluster.cc:1428] Setting key 07a50e6da9aca2b6d2c474819b3b76df
I20260924 10:48:24.371764 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.364085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38147 (local address 127.2.182.1:34757)
0924 10:48:24.364681 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.364719 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.366021 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.366554 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.366560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.366635 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:24.367055 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.367065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.367846 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.367850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.368671 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.368679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.369257 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.369261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.369421 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.370040 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:24.370041 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:24.370077 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:24.370133 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:24.370430 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:24.370442 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:24.370450 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.370578 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.370825 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":117,"threads_started":1}
I20260924 10:48:24.373015 21618 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38147
I20260924 10:48:24.373235 21618 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:24.373795 21618 heartbeater.cc:511] Master 127.2.182.62:38147 requested a full tablet report, sending...
I20260924 10:48:24.373915 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--builtin_ntp_servers=127.2.182.20:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:24.375171 21429 ts_manager.cc:194] Registered new tserver with Master: d67140590beb44af99396eb6d99ad9b4 (127.2.182.1:39625)
I20260924 10:48:24.376264 21429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:34757
W20260924 10:48:24.476022 21621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:24.476284 21621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:24.476349 21621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:24.480984 21621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:24.481052 21621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:24.481134 21621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:24.485819 21621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:24.486788 21621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:24.488394 21621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:24.497587 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.497692 21621 server_base.cc:1034] running on GCE node
W20260924 10:48:24.497587 21629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.497648 21627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.498268 21621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:24.498845 21621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:24.499998 21621 hybrid_clock.cc:648] HybridClock initialized: now 1790246904499973 us; error 37 us; skew 500 ppm
I20260924 10:48:24.501744 21621 webserver.cc:460] Webserver started at http://127.2.182.2:39735/ using document root <none> and password file <none>
I20260924 10:48:24.502254 21621 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:24.502307 21621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:24.502508 21621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:24.504384 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "2d6cf03b38f74a4aa91e6f279a282eda"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "561b1a263c1ae581ad82bb5547ab10e6"
server_key_iv: "f326852309e345e731755e206756471c"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.504971 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "2d6cf03b38f74a4aa91e6f279a282eda"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "561b1a263c1ae581ad82bb5547ab10e6"
server_key_iv: "f326852309e345e731755e206756471c"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.508496 21621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:48:24.510816 21635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.511933 21621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:24.512043 21621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "2d6cf03b38f74a4aa91e6f279a282eda"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "561b1a263c1ae581ad82bb5547ab10e6"
server_key_iv: "f326852309e345e731755e206756471c"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.512137 21621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:24.527431 21621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:24.528019 21621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:24.528214 21621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:24.528769 21621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:24.529709 21621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:24.529783 21621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.529865 21621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:24.529904 21621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.538297 21621 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42797
I20260924 10:48:24.538332 21748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:42797 every 8 connection(s)
I20260924 10:48:24.539098 21621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:24.539979 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21621
I20260924 10:48:24.540067 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:24.540407 2776 external_mini_cluster.cc:1428] Setting key 7c31300c1630cfab87a8917f6d813acc
I20260924 10:48:24.542505 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--builtin_ntp_servers=127.2.182.20:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:24.549399 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.540881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:46807 (local address 127.2.182.62:38147)
0924 10:48:24.541049 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.541055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.542335 (+ 1280us) server_negotiation.cc:407] Connection header received
0924 10:48:24.543858 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.543860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.543902 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.543971 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:24.545091 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.545634 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.547366 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.547565 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.547636 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.548938 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:24.548946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:24.548948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:24.548959 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:24.548984 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:24.548993 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:24.549050 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.549247 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.549293 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:24.550571 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.541275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38147 (local address 127.2.182.2:46807)
0924 10:48:24.542212 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.542249 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.543625 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.544160 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.544171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.544290 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:24.544967 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.544985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.545763 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.545769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.547249 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.547262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.547715 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.547721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.547949 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.548581 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:24.548585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:24.548640 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:24.548723 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:24.549081 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:24.549093 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:24.549101 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.549235 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.549551 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":150,"threads_started":1}
I20260924 10:48:24.551676 21749 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38147
I20260924 10:48:24.551891 21749 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:24.552340 21749 heartbeater.cc:511] Master 127.2.182.62:38147 requested a full tablet report, sending...
I20260924 10:48:24.553355 21429 ts_manager.cc:194] Registered new tserver with Master: 2d6cf03b38f74a4aa91e6f279a282eda (127.2.182.2:42797)
I20260924 10:48:24.554123 21429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:46807
W20260924 10:48:24.648332 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:24.648568 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:24.648605 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:24.652561 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:24.652616 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:24.652669 21752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:24.657445 21752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:24.658407 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:24.659668 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:24.668717 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.668751 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:24.668753 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:24.669070 21752 server_base.cc:1034] running on GCE node
I20260924 10:48:24.669432 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:24.670010 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:24.671164 21752 hybrid_clock.cc:648] HybridClock initialized: now 1790246904671139 us; error 34 us; skew 500 ppm
I20260924 10:48:24.672806 21752 webserver.cc:460] Webserver started at http://127.2.182.3:38583/ using document root <none> and password file <none>
I20260924 10:48:24.673363 21752 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:24.673422 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:24.673666 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:24.675581 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "3a6ad92731a54ab38d4d1183208caa02"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2156f559eae9f3046596fced40a63401"
server_key_iv: "9a7a2506d327bd991d2120a81062035a"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.676117 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "3a6ad92731a54ab38d4d1183208caa02"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2156f559eae9f3046596fced40a63401"
server_key_iv: "9a7a2506d327bd991d2120a81062035a"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.679734 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:48:24.682108 21766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.683280 21752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:24.683393 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "3a6ad92731a54ab38d4d1183208caa02"
format_stamp: "Formatted at 2026-09-24 10:48:24 on dist-test-slave-6lwt"
server_key: "2156f559eae9f3046596fced40a63401"
server_key_iv: "9a7a2506d327bd991d2120a81062035a"
server_key_version: "encryptionkey@0"
I20260924 10:48:24.683491 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:24.696264 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:24.696871 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:24.697098 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:24.697635 21752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:24.698499 21752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:24.698544 21752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.698609 21752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:24.698647 21752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:24.707126 21752 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:46061
I20260924 10:48:24.707154 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:46061 every 8 connection(s)
I20260924 10:48:24.707971 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:48:24.709033 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21752
I20260924 10:48:24.709131 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:24.709436 2776 external_mini_cluster.cc:1428] Setting key 0b7cdf73c0c3d92e4fbcd6c76a8c1e2b
I20260924 10:48:24.717262 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.709829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:42155 (local address 127.2.182.62:38147)
0924 10:48:24.709996 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.710002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.711164 (+ 1162us) server_negotiation.cc:407] Connection header received
0924 10:48:24.712595 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.712598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.712651 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.712727 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:24.713613 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.714157 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.715339 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.715570 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.715679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.716766 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:24.716770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:24.716772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:24.716785 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:24.716810 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:24.716818 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:24.716896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.717068 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.717127 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:48:24.718063 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.710310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:38147 (local address 127.2.182.3:42155)
0924 10:48:24.711041 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.711079 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.712400 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.712881 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.712928 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.713022 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:24.713483 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.713494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.714278 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.714286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.715219 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.715228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.715712 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.715716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.715891 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.716503 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:24.716504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:24.716544 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:24.716605 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:24.716918 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:24.716928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:24.716934 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.717069 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.717336 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260924 10:48:24.719079 21880 heartbeater.cc:347] Connected to a master server at 127.2.182.62:38147
I20260924 10:48:24.719269 21880 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:24.719709 21880 heartbeater.cc:511] Master 127.2.182.62:38147 requested a full tablet report, sending...
I20260924 10:48:24.720602 21429 ts_manager.cc:194] Registered new tserver with Master: 3a6ad92731a54ab38d4d1183208caa02 (127.2.182.3:46061)
I20260924 10:48:24.721194 21429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:42155
I20260924 10:48:24.723272 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:24.728962 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.725126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.2.182.62:38147)
0924 10:48:24.725272 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.725276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.725483 (+ 207us) server_negotiation.cc:407] Connection header received
0924 10:48:24.725599 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.725602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.725649 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.725731 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:24.726013 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.726505 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.727343 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.727564 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.727666 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.728477 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:24.728480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:24.728483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:24.728495 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:24.728518 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:24.728526 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:24.728650 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.728774 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.728808 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:48:24.731441 21429 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:24.734288 21429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:24.747999 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.742730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.2.182.2:42797)
0924 10:48:24.743118 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.743131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.743279 (+ 148us) server_negotiation.cc:407] Connection header received
0924 10:48:24.743370 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.743377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.743572 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.743989 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:24.744127 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.744979 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.746830 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747352 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.747486 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.747676 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.747736 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.747826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260924 10:48:24.748010 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.742524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42797 (local address 127.0.0.1:54106)
0924 10:48:24.743167 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.743180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.743277 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.743720 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.743722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.743736 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:24.743822 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.743830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.745089 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.745095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.746686 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.746697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747477 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.747481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.747581 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.747614 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.747712 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.747827 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":193,"thread_start_us":78,"threads_started":1}
I20260924 10:48:24.748045 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.741792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39625 (local address 127.0.0.1:42512)
0924 10:48:24.742334 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.742383 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.742501 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.743989 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.743992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.744008 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:24.744083 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.744088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.745389 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.745404 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.746533 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.746542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747605 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.747609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.747743 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.747766 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.747829 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.747894 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":17,"thread_start_us":79,"threads_started":1}
I20260924 10:48:24.748265 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.742000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.2.182.1:39625)
0924 10:48:24.743507 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.743521 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.743544 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:48:24.743615 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.743619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.743785 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.744001 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:24.744198 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.745287 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.746660 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747456 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.747765 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.747966 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.748013 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.748106 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":115,"threads_started":1}
I20260924 10:48:24.748437 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.742239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:46061 (local address 127.0.0.1:36150)
0924 10:48:24.742656 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.742669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.742753 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.743598 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.743604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.743629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:24.743774 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.743781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.745166 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.745169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.746692 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.746697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747622 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.747625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.748148 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.748171 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.748236 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.748311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":171,"thread_start_us":100,"threads_started":1}
I20260924 10:48:24.748945 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.742652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.2.182.3:46061)
0924 10:48:24.743102 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.743118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.743142 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:48:24.743217 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.743225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.743455 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.743607 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:24.743892 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.745076 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.746804 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.747507 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.748421 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.748603 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.748651 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.748733 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260924 10:48:24.750942 21683 tablet_service.cc:1467] Processing CreateTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 (DEFAULT_TABLE table=test-table [id=7db40978db7f40039f8ebb99e076905d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:24.751113 21552 tablet_service.cc:1467] Processing CreateTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 (DEFAULT_TABLE table=test-table [id=7db40978db7f40039f8ebb99e076905d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:24.751241 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 (DEFAULT_TABLE table=test-table [id=7db40978db7f40039f8ebb99e076905d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:24.751886 21683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a37b29f8558437b9e3c9665c1b03742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:24.752071 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a37b29f8558437b9e3c9665c1b03742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:24.752071 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a37b29f8558437b9e3c9665c1b03742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:24.758203 21897 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Bootstrap starting.
I20260924 10:48:24.759512 21898 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Bootstrap starting.
I20260924 10:48:24.759615 21899 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Bootstrap starting.
I20260924 10:48:24.760306 21897 tablet_bootstrap.cc:654] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:24.761062 21897 log.cc:824] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:24.761577 21899 tablet_bootstrap.cc:654] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:24.761577 21898 tablet_bootstrap.cc:654] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:24.762372 21898 log.cc:824] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:24.762372 21899 log.cc:824] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:24.763587 21897 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: No bootstrap required, opened a new log
I20260924 10:48:24.763787 21897 ts_tablet_manager.cc:1397] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 10:48:24.764218 21898 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: No bootstrap required, opened a new log
I20260924 10:48:24.764390 21898 ts_tablet_manager.cc:1397] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:48:24.764608 21899 tablet_bootstrap.cc:492] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: No bootstrap required, opened a new log
I20260924 10:48:24.764787 21899 ts_tablet_manager.cc:1397] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:48:24.767006 21897 raft_consensus.cc:348] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.767236 21897 raft_consensus.cc:374] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:24.767288 21897 raft_consensus.cc:729] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6cf03b38f74a4aa91e6f279a282eda, State: Initialized, Role: FOLLOWER
I20260924 10:48:24.767735 21897 consensus_queue.cc:260] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.768560 21898 raft_consensus.cc:348] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.768793 21898 raft_consensus.cc:374] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:24.768851 21898 raft_consensus.cc:729] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67140590beb44af99396eb6d99ad9b4, State: Initialized, Role: FOLLOWER
I20260924 10:48:24.768872 21897 ts_tablet_manager.cc:1428] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 10:48:24.768800 21899 raft_consensus.cc:348] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.769032 21899 raft_consensus.cc:374] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:24.769097 21899 raft_consensus.cc:729] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6ad92731a54ab38d4d1183208caa02, State: Initialized, Role: FOLLOWER
I20260924 10:48:24.769233 21749 heartbeater.cc:503] Master 127.2.182.62:38147 was elected leader, sending a full tablet report...
I20260924 10:48:24.769500 21899 consensus_queue.cc:260] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.769901 21898 consensus_queue.cc:260] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.770098 21880 heartbeater.cc:503] Master 127.2.182.62:38147 was elected leader, sending a full tablet report...
I20260924 10:48:24.770686 21898 ts_tablet_manager.cc:1428] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260924 10:48:24.770988 21618 heartbeater.cc:503] Master 127.2.182.62:38147 was elected leader, sending a full tablet report...
I20260924 10:48:24.771279 21899 ts_tablet_manager.cc:1428] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 10:48:24.789882 21750 tablet.cc:2369] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:24.824934 21904 raft_consensus.cc:482] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:24.825191 21904 raft_consensus.cc:504] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.826296 21904 leader_election.cc:288] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67140590beb44af99396eb6d99ad9b4 (127.2.182.1:39625), 2d6cf03b38f74a4aa91e6f279a282eda (127.2.182.2:42797)
I20260924 10:48:24.830726 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.826520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39625 (local address 127.2.182.3:58875)
0924 10:48:24.826641 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.826657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.826747 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.827101 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.827105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.827124 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:24.827263 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.827272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.828304 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.828307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.829538 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.829549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.830333 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.830336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.830450 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.830487 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.830574 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.830630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:48:24.830840 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.826590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:58875 (local address 127.2.182.1:39625)
0924 10:48:24.826720 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.826724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.826755 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:48:24.826845 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.826849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.826901 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.826984 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:24.827381 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.828202 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.829642 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.830216 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.830322 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.830425 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.830621 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.830696 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:48:24.831055 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.826926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:42067 (local address 127.2.182.2:42797)
0924 10:48:24.827045 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.827049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.827206 (+ 157us) server_negotiation.cc:407] Connection header received
0924 10:48:24.827303 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.827305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.827375 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.827450 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:24.827683 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.828398 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.829750 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.830332 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.830416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.830477 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.830842 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.830941 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:48:24.831437 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.826802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:42797 (local address 127.2.182.3:42067)
0924 10:48:24.827120 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.827132 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:24.827217 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:24.827463 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:24.827465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:24.827495 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:24.827587 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.827590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.828484 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.828487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.829625 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:24.829632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.830602 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.830606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:24.830702 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.830724 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:48:24.831280 (+ 556us) client_negotiation.cc:240] Negotiation successful
0924 10:48:24.831339 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260924 10:48:24.831374 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a37b29f8558437b9e3c9665c1b03742" candidate_uuid: "3a6ad92731a54ab38d4d1183208caa02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67140590beb44af99396eb6d99ad9b4" is_pre_election: true
I20260924 10:48:24.831710 21572 raft_consensus.cc:2454] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6ad92731a54ab38d4d1183208caa02 in term 0.
I20260924 10:48:24.832077 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a37b29f8558437b9e3c9665c1b03742" candidate_uuid: "3a6ad92731a54ab38d4d1183208caa02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" is_pre_election: true
I20260924 10:48:24.832175 21767 leader_election.cc:302] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a6ad92731a54ab38d4d1183208caa02, d67140590beb44af99396eb6d99ad9b4; no voters:
I20260924 10:48:24.832358 21703 raft_consensus.cc:2454] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6ad92731a54ab38d4d1183208caa02 in term 0.
I20260924 10:48:24.832444 21904 raft_consensus.cc:2781] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:24.832527 21904 raft_consensus.cc:482] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:24.832598 21904 raft_consensus.cc:3037] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:24.833776 21904 raft_consensus.cc:504] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.834156 21904 leader_election.cc:288] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [CANDIDATE]: Term 1 election: Requested vote from peers d67140590beb44af99396eb6d99ad9b4 (127.2.182.1:39625), 2d6cf03b38f74a4aa91e6f279a282eda (127.2.182.2:42797)
I20260924 10:48:24.834580 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a37b29f8558437b9e3c9665c1b03742" candidate_uuid: "3a6ad92731a54ab38d4d1183208caa02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6cf03b38f74a4aa91e6f279a282eda"
I20260924 10:48:24.834602 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a37b29f8558437b9e3c9665c1b03742" candidate_uuid: "3a6ad92731a54ab38d4d1183208caa02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67140590beb44af99396eb6d99ad9b4"
I20260924 10:48:24.834702 21572 raft_consensus.cc:3037] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:24.834702 21703 raft_consensus.cc:3037] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:24.835920 21572 raft_consensus.cc:2454] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6ad92731a54ab38d4d1183208caa02 in term 1.
I20260924 10:48:24.835920 21703 raft_consensus.cc:2454] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6ad92731a54ab38d4d1183208caa02 in term 1.
I20260924 10:48:24.836215 21767 leader_election.cc:302] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d6cf03b38f74a4aa91e6f279a282eda, 3a6ad92731a54ab38d4d1183208caa02; no voters:
I20260924 10:48:24.836365 21904 raft_consensus.cc:2781] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:24.836584 21904 raft_consensus.cc:686] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6ad92731a54ab38d4d1183208caa02, State: Running, Role: LEADER
I20260924 10:48:24.836935 21904 consensus_queue.cc:237] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } }
I20260924 10:48:24.839843 21428 catalog_manager.cc:5495] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a6ad92731a54ab38d4d1183208caa02 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "3a6ad92731a54ab38d4d1183208caa02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a6ad92731a54ab38d4d1183208caa02" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46061 } health_report { overall_health: HEALTHY } } }
W20260924 10:48:24.863189 21619 tablet.cc:2369] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:24.873039 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.869485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.2.182.3:46061)
0924 10:48:24.869600 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:24.869605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:24.869619 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:48:24.869724 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:24.869727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:24.869777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:24.869882 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:24.870150 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.870671 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.871540 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:24.871762 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:24.871871 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:24.872200 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:24.872591 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:24.872747 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:24.872829 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:48:24.872863 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:48:24.878433 21572 raft_consensus.cc:1261] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4 [term 1 FOLLOWER]: Refusing update from remote peer 3a6ad92731a54ab38d4d1183208caa02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:24.878440 21703 raft_consensus.cc:1261] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 1 FOLLOWER]: Refusing update from remote peer 3a6ad92731a54ab38d4d1183208caa02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:24.879109 21907 consensus_queue.cc:1035] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67140590beb44af99396eb6d99ad9b4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:24.879247 21904 consensus_queue.cc:1035] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6cf03b38f74a4aa91e6f279a282eda" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:24.887241 21912 mvcc.cc:205] Tried to move back new op lower bound from 7332851322375266304 to 7332851322214612992. Current Snapshot: MvccSnapshot[applied={T|T < 7332851322375266304}]
I20260924 10:48:24.888722 21911 mvcc.cc:205] Tried to move back new op lower bound from 7332851322375266304 to 7332851322214612992. Current Snapshot: MvccSnapshot[applied={T|T < 7332851322375266304 or (T in {7332851322375266304})}]
I20260924 10:48:24.894384 21428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:24.894550 21428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:24.896588 21428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7fb6614bcaf43e3a88d1a8ca87032f1: Sending DeleteTablet for 3 replicas of tablet 6a37b29f8558437b9e3c9665c1b03742
I20260924 10:48:24.897271 21683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:24 UTC) from {username='slave'} at 127.0.0.1:54106
I20260924 10:48:24.897269 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:24 UTC) from {username='slave'} at 127.0.0.1:36150
I20260924 10:48:24.897712 21920 tablet_replica.cc:330] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: stopping tablet replica
I20260924 10:48:24.897722 21552 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a37b29f8558437b9e3c9665c1b03742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:24 UTC) from {username='slave'} at 127.0.0.1:42512
I20260924 10:48:24.898092 21920 raft_consensus.cc:2229] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:24.898142 21922 tablet_replica.cc:330] T 6a37b29f8558437b9e3c9665c1b03742 P d67140590beb44af99396eb6d99ad9b4: stopping tablet replica
I20260924 10:48:24.898355 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21490
I20260924 10:48:24.898993 21920 raft_consensus.cc:2258] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:24.900436 21920 ts_tablet_manager.cc:1905] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:24.903198 21921 tablet_replica.cc:330] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: stopping tablet replica
I20260924 10:48:24.903314 21920 ts_tablet_manager.cc:1918] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:24.903452 21920 log.cc:1196] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-1/wal/wals/6a37b29f8558437b9e3c9665c1b03742
I20260924 10:48:24.903457 21921 raft_consensus.cc:2229] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:24.903812 21921 raft_consensus.cc:2258] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:24.904030 21920 ts_tablet_manager.cc:1939] T 6a37b29f8558437b9e3c9665c1b03742 P 2d6cf03b38f74a4aa91e6f279a282eda: Deleting consensus metadata
I20260924 10:48:24.905203 21414 catalog_manager.cc:4841] TS 2d6cf03b38f74a4aa91e6f279a282eda (127.2.182.2:42797): tablet 6a37b29f8558437b9e3c9665c1b03742 (table test-table [id=7db40978db7f40039f8ebb99e076905d]) successfully deleted
I20260924 10:48:24.905639 21921 ts_tablet_manager.cc:1905] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:48:24.906550 21414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:39625 (error 108)
I20260924 10:48:24.906803 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21621
I20260924 10:48:24.908138 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:24.907885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39625 (local address 127.0.0.1:42526)
0924 10:48:24.907996 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:24.908066 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 10:48:24.908295 21414 catalog_manager.cc:4584] TS d67140590beb44af99396eb6d99ad9b4 (127.2.182.1:39625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a37b29f8558437b9e3c9665c1b03742: Network error: Client connection negotiation failed: client connection to 127.2.182.1:39625: connect: Connection refused (error 111)
I20260924 10:48:24.909507 21921 ts_tablet_manager.cc:1918] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:24.909588 21921 log.cc:1196] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/6a37b29f8558437b9e3c9665c1b03742
I20260924 10:48:24.909853 21921 ts_tablet_manager.cc:1939] T 6a37b29f8558437b9e3c9665c1b03742 P 3a6ad92731a54ab38d4d1183208caa02: Deleting consensus metadata
I20260924 10:48:24.911103 21416 catalog_manager.cc:4841] TS 3a6ad92731a54ab38d4d1183208caa02 (127.2.182.3:46061): tablet 6a37b29f8558437b9e3c9665c1b03742 (table test-table [id=7db40978db7f40039f8ebb99e076905d]) successfully deleted
I20260924 10:48:24.912490 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21752
I20260924 10:48:24.917141 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21399
2026-09-24T10:48:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (992 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:24 dist-test-slave-6lwt krb5kdc[21926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:24 dist-test-slave-6lwt krb5kdc[21926](info): set up 2 sockets
Sep 24 10:48:24 dist-test-slave-6lwt krb5kdc[21926](info): commencing operation
krb5kdc: starting...
W20260924 10:48:25.929224 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:26.047547 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36735
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:26.152657 21939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:26.152945 21939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:26.153049 21939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:26.156919 21939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:26.156991 21939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:26.157024 21939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:26.157080 21939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:26.157119 21939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:26.162067 21939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:26.162976 21939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:26.163947 21939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:26.171720 21945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:26.171705 21944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.172005 21939 server_base.cc:1034] running on GCE node
W20260924 10:48:26.171708 21947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.172593 21939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:26.173676 21939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:26.174880 21939 hybrid_clock.cc:648] HybridClock initialized: now 1790246906174856 us; error 44 us; skew 500 ppm
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:26.177853 21939 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:26.178694 21939 webserver.cc:460] Webserver started at http://127.2.182.62:45805/ using document root <none> and password file <none>
I20260924 10:48:26.179225 21939 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:26.179312 21939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:26.179539 21939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:26.181600 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "3b801b296f7b42ddb8eec166279ecb5e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "7543dc6a53b1c8174fe538f384f57186"
server_key_iv: "83ed4314d20e6ccfdfa937c0dbd4c8bc"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.182145 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "3b801b296f7b42ddb8eec166279ecb5e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "7543dc6a53b1c8174fe538f384f57186"
server_key_iv: "83ed4314d20e6ccfdfa937c0dbd4c8bc"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.185736 21939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:26.187965 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.189153 21939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:26.189288 21939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "3b801b296f7b42ddb8eec166279ecb5e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "7543dc6a53b1c8174fe538f384f57186"
server_key_iv: "83ed4314d20e6ccfdfa937c0dbd4c8bc"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.189409 21939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:26.206701 21939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:26.209503 21939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:26.209702 21939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:26.216615 21939 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36735
I20260924 10:48:26.216637 22006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:36735 every 8 connection(s)
I20260924 10:48:26.217566 21939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:26.220062 22007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:26.223199 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 21939
I20260924 10:48:26.223332 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:26.223649 2776 external_mini_cluster.cc:1428] Setting key 5f69f640799be23d65cf12d9aedf5bac
I20260924 10:48:26.225847 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: Bootstrap starting.
I20260924 10:48:26.227886 22007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:26.228605 22007 log.cc:824] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:26.230265 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: No bootstrap required, opened a new log
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:26.234239 22007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } }
I20260924 10:48:26.234429 22007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:26.234462 22007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b801b296f7b42ddb8eec166279ecb5e, State: Initialized, Role: FOLLOWER
I20260924 10:48:26.234887 22007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } }
I20260924 10:48:26.235034 22007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:26.235117 22007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:26.235232 22007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:26.236018 22007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } }
I20260924 10:48:26.236343 22007 leader_election.cc:302] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b801b296f7b42ddb8eec166279ecb5e; no voters:
I20260924 10:48:26.236665 22007 leader_election.cc:288] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:26.236764 22012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:26.236964 22012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [term 1 LEADER]: Becoming Leader. State: Replica: 3b801b296f7b42ddb8eec166279ecb5e, State: Running, Role: LEADER
I20260924 10:48:26.237248 22012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } }
I20260924 10:48:26.237879 22007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:26.237985 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.225053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.2.182.62:36735)
0924 10:48:26.225499 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.225513 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.225548 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:48:26.226204 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.226223 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.226533 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.226900 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:26.227725 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.228823 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.229719 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.230175 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.230348 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.232981 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:26.232986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:26.232997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:26.233022 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:26.234748 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.235192 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.235200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.235206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.235359 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.235636 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.235659 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.235661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.236037 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:26.236253 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.236540 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.237073 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":142,"threads_started":1}
I20260924 10:48:26.238363 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b801b296f7b42ddb8eec166279ecb5e. Latest consensus state: current_term: 1 leader_uuid: "3b801b296f7b42ddb8eec166279ecb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } } }
I20260924 10:48:26.238479 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:26.238859 22013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b801b296f7b42ddb8eec166279ecb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b801b296f7b42ddb8eec166279ecb5e" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36735 } } }
I20260924 10:48:26.238972 22013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:26.242435 22027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:26.242532 22027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:26.242619 22017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:26.243304 22017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:26.248534 22017 catalog_manager.cc:1347] Generated new cluster ID: 9ed5e8a513d64be9b5523632b319cdd9
I20260924 10:48:26.248595 22017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:26.256699 22017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:26.257615 22017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:26.264519 22017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: Generated new TSK 0
I20260924 10:48:26.265312 22017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
Entry for principal kudu/127.2.182.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab.
I20260924 10:48:26.303902 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:26.408322 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:26.408545 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:26.408583 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:26.412410 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:26.412456 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:26.412508 22033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:26.417073 22033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.1.keytab
--principal=kudu/127.2.182.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:26.417898 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:26.418799 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:26.427706 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.427776 22033 server_base.cc:1034] running on GCE node
W20260924 10:48:26.427721 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:26.428005 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.428383 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:26.428995 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:26.430150 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790246906430122 us; error 40 us; skew 500 ppm
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:26.432739 22033 init.cc:377] Logged in from keytab as kudu/127.2.182.1@KRBTEST.COM (short username kudu)
I20260924 10:48:26.433591 22033 webserver.cc:460] Webserver started at http://127.2.182.1:46623/ using document root <none> and password file <none>
I20260924 10:48:26.434101 22033 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:26.434158 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:26.434376 22033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:26.436259 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "c8140747fd2245c2aee451342e8106cb"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "a40bf93301f2c10a3960817747804966"
server_key_iv: "e60263dc8a6fff0902005f14756c4ecd"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.436784 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "c8140747fd2245c2aee451342e8106cb"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "a40bf93301f2c10a3960817747804966"
server_key_iv: "e60263dc8a6fff0902005f14756c4ecd"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.440321 22033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:48:26.442620 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.443753 22033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:26.443895 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "c8140747fd2245c2aee451342e8106cb"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "a40bf93301f2c10a3960817747804966"
server_key_iv: "e60263dc8a6fff0902005f14756c4ecd"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.444005 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:26.463104 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:26.465885 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:26.466077 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:26.466635 22033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:26.467404 22033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:26.467481 22033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.467571 22033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:26.467620 22033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.476197 22033 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:35023
I20260924 10:48:26.476218 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:35023 every 8 connection(s)
I20260924 10:48:26.477099 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:26.479645 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22033
I20260924 10:48:26.479807 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:26.480132 2776 external_mini_cluster.cc:1428] Setting key 8e21d3192bd8eb20134aab5d6daa634c
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.1@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.2@KRBTEST.COM; defaulting to no policy
I20260924 10:48:26.490937 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.478821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:45903 (local address 127.2.182.62:36735)
0924 10:48:26.478977 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.478981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.479745 (+ 764us) server_negotiation.cc:407] Connection header received
0924 10:48:26.481302 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.481305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.481345 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.481450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:26.482638 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.483190 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.484327 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.484557 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.484685 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.487306 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:26.487310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:26.487312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:26.487334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:26.488769 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.489305 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.489310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.489312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.489392 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.489784 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.489787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.489788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.489965 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:26.490075 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.490632 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.490756 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:48:26.491806 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.479167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36735 (local address 127.2.182.1:45903)
0924 10:48:26.479585 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.479627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.481081 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.481641 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.481651 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.482013 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:26.482482 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.482496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.483318 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.483322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.484191 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.484200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.484721 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.484725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.484912 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.485511 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:26.485527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:26.487139 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:26.488888 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.488931 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.488948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.489203 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.489492 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.489496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.489498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.489674 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.490073 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:26.490080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:26.490324 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.490569 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.490886 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.2@KRBTEST.COM" created.
I20260924 10:48:26.493129 22162 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36735
I20260924 10:48:26.493379 22162 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:26.494009 22162 heartbeater.cc:511] Master 127.2.182.62:36735 requested a full tablet report, sending...
I20260924 10:48:26.495539 21971 ts_manager.cc:194] Registered new tserver with Master: c8140747fd2245c2aee451342e8106cb (127.2.182.1:35023)
I20260924 10:48:26.496644 21971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.1@KRBTEST.COM'} at 127.2.182.1:45903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
Entry for principal kudu/127.2.182.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab.
I20260924 10:48:26.509146 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:26.616466 22167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:26.616747 22167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:26.616789 22167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:26.620787 22167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:26.620841 22167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:26.620957 22167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:26.626092 22167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.2.keytab
--principal=kudu/127.2.182.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:26.627063 22167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:26.628038 22167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:26.637413 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:26.637419 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.637550 22167 server_base.cc:1034] running on GCE node
W20260924 10:48:26.637751 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.638157 22167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:26.638768 22167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:26.639955 22167 hybrid_clock.cc:648] HybridClock initialized: now 1790246906639946 us; error 29 us; skew 500 ppm
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:26.642585 22167 init.cc:377] Logged in from keytab as kudu/127.2.182.2@KRBTEST.COM (short username kudu)
I20260924 10:48:26.643457 22167 webserver.cc:460] Webserver started at http://127.2.182.2:33919/ using document root <none> and password file <none>
I20260924 10:48:26.643971 22167 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:26.644029 22167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:26.644251 22167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:26.646170 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f2b10e74e9b0ba75884be9a0932254cb"
server_key_iv: "b815fc3868e18ebee5d1ee18e07a7ff8"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.646757 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f2b10e74e9b0ba75884be9a0932254cb"
server_key_iv: "b815fc3868e18ebee5d1ee18e07a7ff8"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.650332 22167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 10:48:26.652590 22182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.653766 22167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:26.653853 22167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f2b10e74e9b0ba75884be9a0932254cb"
server_key_iv: "b815fc3868e18ebee5d1ee18e07a7ff8"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.653939 22167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:26.665097 22167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:26.667527 22167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:26.667658 22167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:26.668112 22167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:26.668758 22167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:26.668800 22167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.668833 22167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:26.668856 22167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.677188 22167 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:35193
I20260924 10:48:26.677253 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:35193 every 8 connection(s)
I20260924 10:48:26.678012 22167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:26.685396 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22167
I20260924 10:48:26.685542 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:26.685885 2776 external_mini_cluster.cc:1428] Setting key d89b245ec39a905fa261c38ab9087ee1
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.2@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:26.691460 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.679697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:49401 (local address 127.2.182.62:36735)
0924 10:48:26.679822 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.679826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.680556 (+ 730us) server_negotiation.cc:407] Connection header received
0924 10:48:26.681909 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.681912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.681974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.682100 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:26.683160 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.683684 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.684786 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.685091 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.685210 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.687907 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:26.687914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:26.687919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:26.687950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:26.689448 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.690026 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.690028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.690029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.690094 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.690453 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.690455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.690456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.690607 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:26.690695 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.691262 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.691351 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:48:26.692632 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.680023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36735 (local address 127.2.182.2:49401)
0924 10:48:26.680422 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.680458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.681714 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.682255 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.682262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.682586 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:26.683016 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.683027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.683803 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.683807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.684634 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.684642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.685389 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.685393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.685594 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.686230 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:26.686244 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:26.687752 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:26.689558 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.689566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.689580 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.689923 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.690186 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.690189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.690191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.690365 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.690725 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:26.690733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:26.691032 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.691233 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.691568 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260924 10:48:26.693761 22296 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36735
I20260924 10:48:26.694005 22296 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:26.694520 22296 heartbeater.cc:511] Master 127.2.182.62:36735 requested a full tablet report, sending...
I20260924 10:48:26.695564 21971 ts_manager.cc:194] Registered new tserver with Master: 8d606d2cfbe142cf8caf8d2e30cfb65e (127.2.182.2:35193)
WARNING: no policy specified for kudu/127.2.182.3@KRBTEST.COM; defaulting to no policy
I20260924 10:48:26.696141 21971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.2@KRBTEST.COM'} at 127.2.182.2:49401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
Entry for principal kudu/127.2.182.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab.
I20260924 10:48:26.713922 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:26.819101 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:26.819362 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:26.819434 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:26.823390 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:26.823474 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:26.823581 22301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:26.828730 22301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.3.keytab
--principal=kudu/127.2.182.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:26.829677 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:26.830617 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:26.839968 22301 server_base.cc:1034] running on GCE node
W20260924 10:48:26.839922 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:26.839938 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:26.839907 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:26.840595 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:26.841183 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:26.842332 22301 hybrid_clock.cc:648] HybridClock initialized: now 1790246906842316 us; error 28 us; skew 500 ppm
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:26.844976 22301 init.cc:377] Logged in from keytab as kudu/127.2.182.3@KRBTEST.COM (short username kudu)
I20260924 10:48:26.845770 22301 webserver.cc:460] Webserver started at http://127.2.182.3:38965/ using document root <none> and password file <none>
I20260924 10:48:26.846289 22301 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:26.846374 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:26.846588 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:26.848671 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "86afe56d341540b5b7985c4bf3f5d685"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f7ef248cdfe8c1753b1a84691f71a91d"
server_key_iv: "6c4f84ebd8bf77b3f5960431579501b0"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.849222 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "86afe56d341540b5b7985c4bf3f5d685"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f7ef248cdfe8c1753b1a84691f71a91d"
server_key_iv: "6c4f84ebd8bf77b3f5960431579501b0"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.852494 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:26.854687 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.855773 22301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:48:26.855898 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "86afe56d341540b5b7985c4bf3f5d685"
format_stamp: "Formatted at 2026-09-24 10:48:26 on dist-test-slave-6lwt"
server_key: "f7ef248cdfe8c1753b1a84691f71a91d"
server_key_iv: "6c4f84ebd8bf77b3f5960431579501b0"
server_key_version: "encryptionkey@0"
I20260924 10:48:26.856005 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:26.876241 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:26.879446 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:26.879634 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:26.880190 22301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:26.880988 22301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:26.881057 22301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.881129 22301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:26.881172 22301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:26.889752 22301 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:39063
I20260924 10:48:26.889776 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:39063 every 8 connection(s)
I20260924 10:48:26.890570 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:48:26 dist-test-slave-6lwt krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246906, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.3@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:26.899417 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22301
I20260924 10:48:26.899550 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:26.899825 2776 external_mini_cluster.cc:1428] Setting key ddc50ea6f5c2eb5f1130ae43355b8337
I20260924 10:48:26.903748 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.892283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:37281 (local address 127.2.182.62:36735)
0924 10:48:26.892448 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.892452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.893309 (+ 857us) server_negotiation.cc:407] Connection header received
0924 10:48:26.894499 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.894502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.894548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.894622 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:26.895692 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.896206 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.897337 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.897544 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.897621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.900070 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:26.900074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:26.900076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:26.900103 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:26.901819 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.902289 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.902292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.902294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.902378 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.902713 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.902718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.902720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.902906 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:26.903022 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.903465 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.903586 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:48:26.904457 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.892734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:36735 (local address 127.2.182.3:37281)
0924 10:48:26.893165 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.893204 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.894316 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.894774 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.894780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.895117 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:26.895565 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.895577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.896305 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.896308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.897223 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.897232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.897662 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.897665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.897791 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.898376 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:26.898392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:26.899878 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:26.901912 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.901917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.901930 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.902171 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.902467 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:26.902472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:26.902474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:26.902618 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:26.903019 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:26.903025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:26.903245 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.903425 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.903716 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260924 10:48:26.905483 22430 heartbeater.cc:347] Connected to a master server at 127.2.182.62:36735
I20260924 10:48:26.905676 22430 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:26.906071 22430 heartbeater.cc:511] Master 127.2.182.62:36735 requested a full tablet report, sending...
I20260924 10:48:26.906949 21971 ts_manager.cc:194] Registered new tserver with Master: 86afe56d341540b5b7985c4bf3f5d685 (127.2.182.3:39063)
I20260924 10:48:26.907447 21971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.3@KRBTEST.COM'} at 127.2.182.3:37281
I20260924 10:48:26.913427 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:26.921795 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.915200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.2.182.62:36735)
0924 10:48:26.915285 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.915288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.915514 (+ 226us) server_negotiation.cc:407] Connection header received
0924 10:48:26.915598 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.915600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.915632 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.915693 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:26.916308 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.916777 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.917781 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.917975 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.918049 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.919332 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:26.919334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:26.919336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:26.919351 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:26.920482 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.920831 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.920833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.920834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.920942 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:26.921179 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:26.921181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:26.921182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:26.921360 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:26.921457 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.921613 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.921693 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 10:48:26.924228 21971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:26.927107 21971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:26.941256 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.935682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.2.182.3:39063)
0924 10:48:26.936329 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.936342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.936366 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:48:26.936454 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.936459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.936635 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.936783 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:26.937238 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938402 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.939733 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940551 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940684 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.940831 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.941003 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.941094 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":80,"threads_started":1}
I20260924 10:48:26.941254 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.934958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:35193 (local address 127.0.0.1:52608)
0924 10:48:26.935450 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.935493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.935610 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.936454 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.936460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.936483 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:26.936612 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.936619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938352 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.938355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.939731 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.939741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940751 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.940838 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.940871 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.940951 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.941111 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":46,"thread_start_us":85,"threads_started":1}
I20260924 10:48:26.941431 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.935158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:39063 (local address 127.0.0.1:32768)
0924 10:48:26.936008 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.936020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.936096 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.936786 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.936789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.936805 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:26.936900 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.937143 (+ 243us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938562 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.938565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.939611 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.939620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940688 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.940793 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.940827 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.940973 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.941319 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260924 10:48:26.941505 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.935129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.2.182.2:35193)
0924 10:48:26.935935 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.935948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.935971 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:48:26.936049 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.936057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.936255 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.936781 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:26.936957 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938228 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.939857 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940626 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.941071 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.941192 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.941237 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.941333 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":120,"threads_started":1}
I20260924 10:48:26.941255 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.935578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:35023 (local address 127.0.0.1:58628)
0924 10:48:26.936150 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:26.936161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:26.936241 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:26.937001 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:26.937003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:26.937021 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:26.937092 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.937097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938253 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.938259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.939908 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:26.939914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940897 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940921 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:26.941021 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.941049 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:48:26.941111 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:48:26.941163 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":50,"thread_start_us":98,"threads_started":1}
I20260924 10:48:26.942003 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:26.935745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.2.182.1:35023)
0924 10:48:26.936511 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:26.936521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:26.936564 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:48:26.936623 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:26.936626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:26.936833 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:26.937015 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:26.937173 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.938128 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940022 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:26.940582 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:26.940703 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:26.940835 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:26.941755 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 10:48:26.941857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":94,"threads_started":1}
I20260924 10:48:26.944207 22364 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c (DEFAULT_TABLE table=test-table [id=7e97fc01aafc40cbaa237dc605b079a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:26.944310 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c (DEFAULT_TABLE table=test-table [id=7e97fc01aafc40cbaa237dc605b079a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:26.944358 22096 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c (DEFAULT_TABLE table=test-table [id=7e97fc01aafc40cbaa237dc605b079a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:26.945223 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:26.945273 22096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:26.945273 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:26.951701 22447 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Bootstrap starting.
I20260924 10:48:26.952297 22449 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Bootstrap starting.
I20260924 10:48:26.953220 22448 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Bootstrap starting.
I20260924 10:48:26.953742 22447 tablet_bootstrap.cc:654] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:26.954258 22449 tablet_bootstrap.cc:654] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:26.954754 22447 log.cc:824] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:26.955173 22448 tablet_bootstrap.cc:654] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:26.955379 22449 log.cc:824] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:26.956224 22448 log.cc:824] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:26.957082 22447 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: No bootstrap required, opened a new log
I20260924 10:48:26.957327 22447 ts_tablet_manager.cc:1397] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:48:26.957355 22449 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: No bootstrap required, opened a new log
I20260924 10:48:26.957505 22449 ts_tablet_manager.cc:1397] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260924 10:48:26.958217 22448 tablet_bootstrap.cc:492] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: No bootstrap required, opened a new log
I20260924 10:48:26.958415 22448 ts_tablet_manager.cc:1397] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:48:26.960456 22447 raft_consensus.cc:348] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.960740 22447 raft_consensus.cc:374] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:26.960795 22447 raft_consensus.cc:729] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86afe56d341540b5b7985c4bf3f5d685, State: Initialized, Role: FOLLOWER
I20260924 10:48:26.961223 22447 consensus_queue.cc:260] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.961159 22449 raft_consensus.cc:348] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.961403 22449 raft_consensus.cc:374] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:26.961448 22449 raft_consensus.cc:729] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d606d2cfbe142cf8caf8d2e30cfb65e, State: Initialized, Role: FOLLOWER
I20260924 10:48:26.961805 22430 heartbeater.cc:503] Master 127.2.182.62:36735 was elected leader, sending a full tablet report...
I20260924 10:48:26.961834 22449 consensus_queue.cc:260] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.962127 22448 raft_consensus.cc:348] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.962343 22448 raft_consensus.cc:374] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:26.962368 22447 ts_tablet_manager.cc:1428] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:26.962385 22448 raft_consensus.cc:729] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8140747fd2245c2aee451342e8106cb, State: Initialized, Role: FOLLOWER
I20260924 10:48:26.962698 22449 ts_tablet_manager.cc:1428] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:26.962980 22296 heartbeater.cc:503] Master 127.2.182.62:36735 was elected leader, sending a full tablet report...
I20260924 10:48:26.962790 22448 consensus_queue.cc:260] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:26.964430 22162 heartbeater.cc:503] Master 127.2.182.62:36735 was elected leader, sending a full tablet report...
I20260924 10:48:26.964567 22448 ts_tablet_manager.cc:1428] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260924 10:48:26.978111 22163 tablet.cc:2369] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:48:27.141491 22431 tablet.cc:2369] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:27.145612 22453 raft_consensus.cc:482] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:27.145798 22453 raft_consensus.cc:504] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:27.146992 22453 leader_election.cc:288] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d606d2cfbe142cf8caf8d2e30cfb65e (127.2.182.2:35193), c8140747fd2245c2aee451342e8106cb (127.2.182.1:35023)
I20260924 10:48:27.150754 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.147369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:46173 (local address 127.2.182.2:35193)
0924 10:48:27.147522 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.147525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.147536 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:48:27.147579 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.147582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.147620 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.147712 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:27.147989 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.148669 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.149761 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.150180 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.150296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.150355 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.150556 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.150640 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:48:27.150753 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.147178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:35193 (local address 127.2.182.3:46173)
0924 10:48:27.147288 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.147304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:27.147395 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:27.147726 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:27.147729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:27.147746 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:27.147866 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.147874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.148779 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.148782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.149648 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.149656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.150297 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.150300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.150407 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.150451 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 10:48:27.150528 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:48:27.150601 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":34}
I20260924 10:48:27.151002 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.147315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:39317 (local address 127.2.182.1:35023)
0924 10:48:27.147450 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.147455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.147696 (+ 241us) server_negotiation.cc:407] Connection header received
0924 10:48:27.147858 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.147860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.147900 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.147994 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:27.148226 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.148846 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.149933 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.150300 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.150412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.150488 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.150764 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.150851 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:48:27.151401 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.147242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:35023 (local address 127.2.182.3:39317)
0924 10:48:27.147605 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.147615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:27.147735 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:27.147994 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:27.147996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:27.148011 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:27.148118 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.148124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.148998 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.149002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.149799 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.149808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.150413 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.150417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.150594 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.150624 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:48:27.151142 (+ 518us) client_negotiation.cc:240] Negotiation successful
0924 10:48:27.151227 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":67,"thread_start_us":112,"threads_started":1}
I20260924 10:48:27.151424 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb2dac8a13d4e25b9a2234e73ae9f2c" candidate_uuid: "86afe56d341540b5b7985c4bf3f5d685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" is_pre_election: true
I20260924 10:48:27.151729 22250 raft_consensus.cc:2454] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86afe56d341540b5b7985c4bf3f5d685 in term 0.
I20260924 10:48:27.152096 22116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb2dac8a13d4e25b9a2234e73ae9f2c" candidate_uuid: "86afe56d341540b5b7985c4bf3f5d685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8140747fd2245c2aee451342e8106cb" is_pre_election: true
I20260924 10:48:27.152284 22318 leader_election.cc:302] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86afe56d341540b5b7985c4bf3f5d685, 8d606d2cfbe142cf8caf8d2e30cfb65e; no voters:
I20260924 10:48:27.152355 22116 raft_consensus.cc:2454] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86afe56d341540b5b7985c4bf3f5d685 in term 0.
I20260924 10:48:27.152577 22453 raft_consensus.cc:2781] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:27.152683 22453 raft_consensus.cc:482] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:27.152735 22453 raft_consensus.cc:3037] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:27.154276 22453 raft_consensus.cc:504] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:27.154681 22453 leader_election.cc:288] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [CANDIDATE]: Term 1 election: Requested vote from peers 8d606d2cfbe142cf8caf8d2e30cfb65e (127.2.182.2:35193), c8140747fd2245c2aee451342e8106cb (127.2.182.1:35023)
I20260924 10:48:27.155015 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb2dac8a13d4e25b9a2234e73ae9f2c" candidate_uuid: "86afe56d341540b5b7985c4bf3f5d685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e"
I20260924 10:48:27.155035 22116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb2dac8a13d4e25b9a2234e73ae9f2c" candidate_uuid: "86afe56d341540b5b7985c4bf3f5d685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8140747fd2245c2aee451342e8106cb"
I20260924 10:48:27.155136 22250 raft_consensus.cc:3037] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:27.155136 22116 raft_consensus.cc:3037] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:27.156327 22250 raft_consensus.cc:2454] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86afe56d341540b5b7985c4bf3f5d685 in term 1.
I20260924 10:48:27.156327 22116 raft_consensus.cc:2454] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86afe56d341540b5b7985c4bf3f5d685 in term 1.
I20260924 10:48:27.156646 22318 leader_election.cc:302] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 86afe56d341540b5b7985c4bf3f5d685, 8d606d2cfbe142cf8caf8d2e30cfb65e, c8140747fd2245c2aee451342e8106cb; no voters:
I20260924 10:48:27.157757 22453 raft_consensus.cc:2781] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:27.158010 22453 raft_consensus.cc:686] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 1 LEADER]: Becoming Leader. State: Replica: 86afe56d341540b5b7985c4bf3f5d685, State: Running, Role: LEADER
I20260924 10:48:27.158545 22453 consensus_queue.cc:237] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } }
I20260924 10:48:27.161561 21968 catalog_manager.cc:5495] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86afe56d341540b5b7985c4bf3f5d685 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "86afe56d341540b5b7985c4bf3f5d685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86afe56d341540b5b7985c4bf3f5d685" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 } health_report { overall_health: UNKNOWN } } }
I20260924 10:48:27.172523 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.169077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.2.182.3:39063)
0924 10:48:27.169189 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.169193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.169280 (+ 87us) server_negotiation.cc:407] Connection header received
0924 10:48:27.169457 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.169464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.169514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.169614 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:27.169845 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.170316 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.171161 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.171375 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.171477 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.171784 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:27.172166 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:27.172292 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.172331 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.172379 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:48:27.177650 22116 raft_consensus.cc:1261] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 1 FOLLOWER]: Refusing update from remote peer 86afe56d341540b5b7985c4bf3f5d685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:27.177651 22250 raft_consensus.cc:1261] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 1 FOLLOWER]: Refusing update from remote peer 86afe56d341540b5b7985c4bf3f5d685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:27.178385 22453 consensus_queue.cc:1035] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8140747fd2245c2aee451342e8106cb" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:27.178536 22457 consensus_queue.cc:1035] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d606d2cfbe142cf8caf8d2e30cfb65e" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 35193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:48:27.179072 22297 tablet.cc:2369] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:27.185269 22462 mvcc.cc:205] Tried to move back new op lower bound from 7332851331793301504 to 7332851331723816960. Current Snapshot: MvccSnapshot[applied={T|T < 7332851331793301504}]
I20260924 10:48:27.192380 21968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:27.192519 21968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:27.194748 21968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b801b296f7b42ddb8eec166279ecb5e: Sending DeleteTablet for 3 replicas of tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c
I20260924 10:48:27.195348 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:27 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:52608
I20260924 10:48:27.195456 22364 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:27 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:32768
I20260924 10:48:27.195538 22096 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:27 UTC) from {username='kudu', principal='kudu/127.2.182.62@KRBTEST.COM'} at 127.0.0.1:58628
I20260924 10:48:27.195849 22469 tablet_replica.cc:330] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: stopping tablet replica
I20260924 10:48:27.195904 22471 tablet_replica.cc:330] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb: stopping tablet replica
I20260924 10:48:27.196178 22471 raft_consensus.cc:2229] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:27.196278 22470 tablet_replica.cc:330] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: stopping tablet replica
I20260924 10:48:27.196465 22471 raft_consensus.cc:2258] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P c8140747fd2245c2aee451342e8106cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:27.196496 22470 raft_consensus.cc:2229] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:27.196751 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22033
I20260924 10:48:27.196785 22470 raft_consensus.cc:2258] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:27.200518 22470 ts_tablet_manager.cc:1905] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:27.201541 22469 raft_consensus.cc:2229] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:27.201836 22469 raft_consensus.cc:2258] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:27.203181 22469 ts_tablet_manager.cc:1905] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 8d606d2cfbe142cf8caf8d2e30cfb65e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:27.203857 22470 ts_tablet_manager.cc:1918] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 10:48:27.203872 21958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:35023 (error 108)
I20260924 10:48:27.203955 22470 log.cc:1196] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/2cb2dac8a13d4e25b9a2234e73ae9f2c
I20260924 10:48:27.204309 22470 ts_tablet_manager.cc:1939] T 2cb2dac8a13d4e25b9a2234e73ae9f2c P 86afe56d341540b5b7985c4bf3f5d685: Deleting consensus metadata
I20260924 10:48:27.204701 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22167
I20260924 10:48:27.205475 21957 catalog_manager.cc:4841] TS 86afe56d341540b5b7985c4bf3f5d685 (127.2.182.3:39063): tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c (table test-table [id=7e97fc01aafc40cbaa237dc605b079a2]) successfully deleted
I20260924 10:48:27.210156 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.209885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:35023 (local address 127.0.0.1:58640)
0924 10:48:27.210016 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.210080 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:35023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:48:27.210305 21958 catalog_manager.cc:4584] TS c8140747fd2245c2aee451342e8106cb (127.2.182.1:35023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cb2dac8a13d4e25b9a2234e73ae9f2c: Network error: Client connection negotiation failed: client connection to 127.2.182.1:35023: connect: Connection refused (error 111)
I20260924 10:48:27.211146 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22301
I20260924 10:48:27.212172 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.211351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:35193 (local address 127.0.0.1:52624)
0924 10:48:27.212076 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.212116 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:35193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667}
I20260924 10:48:27.216991 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 21939
2026-09-24T10:48:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2300 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:27Z Disabled control of system clock
I20260924 10:48:27.245200 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46233
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:27.348261 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:27.348474 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:27.348505 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:27.352391 22476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:27.352435 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:27.352449 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:27.352463 22476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:27.352476 22476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:27.356873 22476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:27.357800 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:27.359169 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:27.367012 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.367084 22476 server_base.cc:1034] running on GCE node
W20260924 10:48:27.367033 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.367002 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.367743 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:27.368709 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:27.369864 22476 hybrid_clock.cc:648] HybridClock initialized: now 1790246907369834 us; error 41 us; skew 500 ppm
I20260924 10:48:27.371628 22476 webserver.cc:460] Webserver started at http://127.2.182.62:37099/ using document root <none> and password file <none>
I20260924 10:48:27.372179 22476 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:27.372234 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:27.372454 22476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:27.374444 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "913827fda1374ab4b801810c4d95b93d"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "3339e9a18be6800989c9789290e98ee5"
server_key_iv: "d2d3725f32e7b0e21a400a75a0514142"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.374989 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "913827fda1374ab4b801810c4d95b93d"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "3339e9a18be6800989c9789290e98ee5"
server_key_iv: "d2d3725f32e7b0e21a400a75a0514142"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.378607 22476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:27.381073 22490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.382257 22476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:27.382434 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "913827fda1374ab4b801810c4d95b93d"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "3339e9a18be6800989c9789290e98ee5"
server_key_iv: "d2d3725f32e7b0e21a400a75a0514142"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.382541 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:27.395421 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:27.396010 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:27.396199 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:27.402918 22476 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:46233
I20260924 10:48:27.402935 22542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:46233 every 8 connection(s)
I20260924 10:48:27.403774 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:27.406580 22543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:27.410151 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22476
I20260924 10:48:27.410292 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:27.410558 2776 external_mini_cluster.cc:1428] Setting key 1913c38ba1ccaa23a3e352b8bac3a4cf
I20260924 10:48:27.411980 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: Bootstrap starting.
I20260924 10:48:27.414065 22543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:27.414721 22543 log.cc:824] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:27.416375 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: No bootstrap required, opened a new log
I20260924 10:48:27.418943 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.411918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.2.182.62:46233)
0924 10:48:27.412482 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.412496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.412528 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:48:27.413151 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.413167 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.413529 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.413768 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:27.414108 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.415095 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.415968 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.416423 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.416594 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.417457 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:27.417462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:27.417472 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:27.417796 (+ 324us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:27.417849 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:27.417867 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:27.417966 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.418121 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.418349 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":140,"threads_started":1}
I20260924 10:48:27.419862 22543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } }
I20260924 10:48:27.420090 22543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:27.420135 22543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913827fda1374ab4b801810c4d95b93d, State: Initialized, Role: FOLLOWER
I20260924 10:48:27.420538 22543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } }
I20260924 10:48:27.420679 22543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:27.420787 22543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:27.420883 22543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:27.421660 22543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } }
I20260924 10:48:27.421941 22543 leader_election.cc:302] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913827fda1374ab4b801810c4d95b93d; no voters:
I20260924 10:48:27.422149 22543 leader_election.cc:288] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:27.422254 22548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:27.422478 22548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [term 1 LEADER]: Becoming Leader. State: Replica: 913827fda1374ab4b801810c4d95b93d, State: Running, Role: LEADER
I20260924 10:48:27.422760 22548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } }
I20260924 10:48:27.423236 22543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:27.424335 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913827fda1374ab4b801810c4d95b93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } } }
I20260924 10:48:27.424463 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:27.424345 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913827fda1374ab4b801810c4d95b93d. Latest consensus state: current_term: 1 leader_uuid: "913827fda1374ab4b801810c4d95b93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913827fda1374ab4b801810c4d95b93d" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46233 } } }
I20260924 10:48:27.424543 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:27.424925 22557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:27.427783 22557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:27.433578 22557 catalog_manager.cc:1347] Generated new cluster ID: ebe42609228d4d7d9dc573ef17479630
I20260924 10:48:27.433653 22557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:27.454073 22557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:27.454864 22557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:27.461956 22557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: Generated new TSK 0
I20260924 10:48:27.462612 22557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:27.471010 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--builtin_ntp_servers=127.2.182.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:48:27.590829 22567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:27.591087 22567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:27.591152 22567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:27.595153 22567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:27.595225 22567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:27.595306 22567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20260924 10:48:27.599985 22567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:27.601096 22567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:27.602447 22567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:27.611778 22573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.611780 22572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.611790 22575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.612001 22567 server_base.cc:1034] running on GCE node
I20260924 10:48:27.612519 22567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:27.613245 22567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:27.614415 22567 hybrid_clock.cc:648] HybridClock initialized: now 1790246907614371 us; error 26 us; skew 500 ppm
I20260924 10:48:27.616110 22567 webserver.cc:460] Webserver started at http://127.2.182.1:34891/ using document root <none> and password file <none>
I20260924 10:48:27.616673 22567 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:27.616731 22567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:27.617061 22567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:27.619033 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data/instance:
uuid: "7c30db5d936143a9b8a6fdda01128314"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "5f52763a6b4ba8466bdcbdc20b395d80"
server_key_iv: "08da952193a086789b4d9831022f27db"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.619572 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "7c30db5d936143a9b8a6fdda01128314"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "5f52763a6b4ba8466bdcbdc20b395d80"
server_key_iv: "08da952193a086789b4d9831022f27db"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.623143 22567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:48:27.625772 22581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.626893 22567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:27.627030 22567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal
uuid: "7c30db5d936143a9b8a6fdda01128314"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "5f52763a6b4ba8466bdcbdc20b395d80"
server_key_iv: "08da952193a086789b4d9831022f27db"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.627143 22567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:27.653136 22567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:27.653831 22567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:27.654039 22567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:27.654615 22567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:27.655539 22567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:27.655583 22567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.655651 22567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:27.655681 22567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.664239 22567 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:39159
I20260924 10:48:27.664264 22694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.1:39159 every 8 connection(s)
I20260924 10:48:27.665174 22567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/data/info.pb
I20260924 10:48:27.668136 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22567
I20260924 10:48:27.668256 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-0/wal/instance
I20260924 10:48:27.668581 2776 external_mini_cluster.cc:1428] Setting key 75785c104161826c41f697e8211377aa
I20260924 10:48:27.670809 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--builtin_ntp_servers=127.2.182.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:27.674973 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.666903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.1:48503 (local address 127.2.182.62:46233)
0924 10:48:27.667038 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.667042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.668151 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 10:48:27.669741 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.669748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.669785 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.669854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:27.670977 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.671525 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.672845 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.673192 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.673276 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.674526 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:27.674529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:27.674531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:27.674542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:27.674559 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:27.674566 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:27.674624 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.674814 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.674861 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:48:27.676249 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.667283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46233 (local address 127.2.182.1:48503)
0924 10:48:27.667981 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.668037 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:27.669545 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:27.670035 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:27.670042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:27.670161 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:27.670815 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.670837 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.671660 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.671667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.672722 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.672732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.673346 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.673353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.673586 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.674229 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:27.674234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:27.674286 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:27.674369 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:27.674653 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:27.674664 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:27.674672 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:48:27.674815 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:48:27.675151 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260924 10:48:27.677592 22695 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46233
I20260924 10:48:27.677810 22695 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:27.678303 22695 heartbeater.cc:511] Master 127.2.182.62:46233 requested a full tablet report, sending...
I20260924 10:48:27.679744 22507 ts_manager.cc:194] Registered new tserver with Master: 7c30db5d936143a9b8a6fdda01128314 (127.2.182.1:39159)
I20260924 10:48:27.680842 22507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:48503
W20260924 10:48:27.777812 22698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:27.778086 22698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:27.778158 22698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:27.782053 22698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:27.782133 22698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:27.782224 22698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20260924 10:48:27.786870 22698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:27.787827 22698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:27.789165 22698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:27.798246 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.798257 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.798456 22698 server_base.cc:1034] running on GCE node
W20260924 10:48:27.798254 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.799065 22698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:27.799736 22698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:27.800947 22698 hybrid_clock.cc:648] HybridClock initialized: now 1790246907800931 us; error 31 us; skew 500 ppm
I20260924 10:48:27.802809 22698 webserver.cc:460] Webserver started at http://127.2.182.2:39815/ using document root <none> and password file <none>
I20260924 10:48:27.803403 22698 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:27.803496 22698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:27.803706 22698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:27.805805 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data/instance:
uuid: "918eea415fa54354bdb2355f74d7cfd0"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "0f5af2e75843cf5536b9ac2e3f3f9242"
server_key_iv: "f901d4e789a2f3c0e65722b21553d2c8"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.806349 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "918eea415fa54354bdb2355f74d7cfd0"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "0f5af2e75843cf5536b9ac2e3f3f9242"
server_key_iv: "f901d4e789a2f3c0e65722b21553d2c8"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.810175 22698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:48:27.812548 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.813753 22698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:27.813886 22698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal
uuid: "918eea415fa54354bdb2355f74d7cfd0"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "0f5af2e75843cf5536b9ac2e3f3f9242"
server_key_iv: "f901d4e789a2f3c0e65722b21553d2c8"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.813997 22698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:27.825037 22698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:27.825609 22698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:27.825783 22698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:27.826290 22698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:27.827113 22698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:27.827181 22698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.827242 22698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:27.827288 22698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.835587 22698 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:39175
I20260924 10:48:27.835611 22825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.2:39175 every 8 connection(s)
I20260924 10:48:27.836400 22698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/data/info.pb
I20260924 10:48:27.837177 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22698
I20260924 10:48:27.837316 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-1/wal/instance
I20260924 10:48:27.837626 2776 external_mini_cluster.cc:1428] Setting key 2570d8cd7269e57f1c9386041515b868
I20260924 10:48:27.839640 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--builtin_ntp_servers=127.2.182.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:48:27.846648 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.838200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:57819 (local address 127.2.182.62:46233)
0924 10:48:27.838329 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:27.838333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:27.839483 (+ 1150us) server_negotiation.cc:407] Connection header received
0924 10:48:27.841005 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:27.841009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:27.841091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:27.841189 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:27.842365 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.843289 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.844487 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.844946 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.845064 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.846089 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:27.846096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:27.846101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:27.846117 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:27.846143 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:27.846156 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:27.846238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:27.846385 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:48:27.846452 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:48:27.847332 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:27.838633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46233 (local address 127.2.182.2:57819)
0924 10:48:27.839305 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:27.839353 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:27.840700 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:27.841449 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:27.841458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:27.841567 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:27.842220 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.842234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.843421 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.843424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.844360 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:27.844369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:27.845074 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:27.845077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:27.845231 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:27.845819 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:27.845820 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:27.845873 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:27.845929 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:27.846251 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:27.846257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:27.846261 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:48:27.846346 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:48:27.846538 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260924 10:48:27.848393 22826 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46233
I20260924 10:48:27.848600 22826 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:27.849141 22826 heartbeater.cc:511] Master 127.2.182.62:46233 requested a full tablet report, sending...
I20260924 10:48:27.850090 22507 ts_manager.cc:194] Registered new tserver with Master: 918eea415fa54354bdb2355f74d7cfd0 (127.2.182.2:39175)
I20260924 10:48:27.850638 22507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:57819
W20260924 10:48:27.944551 22829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:27.944787 22829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:27.944868 22829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:27.948571 22829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:27.948639 22829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:27.948724 22829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20260924 10:48:27.953182 22829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:27.954051 22829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:27.955318 22829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:27.964151 22837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.964145 22834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:27.964267 22835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:27.964521 22829 server_base.cc:1034] running on GCE node
I20260924 10:48:27.964934 22829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:27.965562 22829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:27.966775 22829 hybrid_clock.cc:648] HybridClock initialized: now 1790246907966760 us; error 25 us; skew 500 ppm
I20260924 10:48:27.968552 22829 webserver.cc:460] Webserver started at http://127.2.182.3:43153/ using document root <none> and password file <none>
I20260924 10:48:27.969146 22829 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:27.969233 22829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:27.969451 22829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:27.971331 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data/instance:
uuid: "b80b294e90294817b664b99b1c95d283"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "68b7034e20f85f4c6b07676aa6054044"
server_key_iv: "2c5049a72549758814655141ca071f80"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.971871 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "b80b294e90294817b664b99b1c95d283"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "68b7034e20f85f4c6b07676aa6054044"
server_key_iv: "2c5049a72549758814655141ca071f80"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.975425 22829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:27.977710 22843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.978822 22829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:27.978957 22829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal
uuid: "b80b294e90294817b664b99b1c95d283"
format_stamp: "Formatted at 2026-09-24 10:48:27 on dist-test-slave-6lwt"
server_key: "68b7034e20f85f4c6b07676aa6054044"
server_key_iv: "2c5049a72549758814655141ca071f80"
server_key_version: "encryptionkey@0"
I20260924 10:48:27.979061 22829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:27.988183 22829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:27.988754 22829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:27.988991 22829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:27.989498 22829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:48:27.990302 22829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:48:27.990370 22829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.990446 22829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:48:27.990482 22829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:27.999070 22829 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:36357
I20260924 10:48:27.999101 22956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.3:36357 every 8 connection(s)
I20260924 10:48:27.999894 22829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/data/info.pb
I20260924 10:48:28.006213 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 22829
I20260924 10:48:28.006314 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal/instance
I20260924 10:48:28.006629 2776 external_mini_cluster.cc:1428] Setting key 429d29640ad27566412d4d408c2f6a6e
I20260924 10:48:28.008616 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.001520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.3:53473 (local address 127.2.182.62:46233)
0924 10:48:28.001679 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.001684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.002653 (+ 969us) server_negotiation.cc:407] Connection header received
0924 10:48:28.003938 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.003941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.003986 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.004061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:28.004828 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.005382 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.006662 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.006921 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.007027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.008137 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:28.008140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:28.008142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:28.008151 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:28.008169 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:28.008177 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:28.008233 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.008400 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.008484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:48:28.009438 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.001916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:46233 (local address 127.2.182.3:53473)
0924 10:48:28.002510 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.002546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.003760 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.004214 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.004221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.004292 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:28.004716 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.004727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.005483 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.005486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.006548 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.006560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.007064 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.007067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.007249 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.007873 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:48:28.007874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:28.007941 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:48:28.008013 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:28.008254 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:28.008262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:28.008267 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.008399 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.008664 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260924 10:48:28.010356 22957 heartbeater.cc:347] Connected to a master server at 127.2.182.62:46233
I20260924 10:48:28.010576 22957 heartbeater.cc:464] Registering TS with master...
I20260924 10:48:28.011039 22957 heartbeater.cc:511] Master 127.2.182.62:46233 requested a full tablet report, sending...
I20260924 10:48:28.011937 22507 ts_manager.cc:194] Registered new tserver with Master: b80b294e90294817b664b99b1c95d283 (127.2.182.3:36357)
I20260924 10:48:28.012472 22507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:53473
I20260924 10:48:28.020138 2776 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:48:28.025676 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.022012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.2.182.62:46233)
0924 10:48:28.022150 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.022155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.022313 (+ 158us) server_negotiation.cc:407] Connection header received
0924 10:48:28.022459 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.022462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.022502 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.022586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:28.022831 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.023351 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.024219 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.024452 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.024555 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.025267 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:28.025271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:28.025273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:28.025284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:28.025301 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:28.025309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:28.025393 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.025503 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.025550 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:48:28.028023 22507 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:48:28.031059 22507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:48:28.044365 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.038868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.2.182.2:39175)
0924 10:48:28.039238 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.039249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.039293 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:48:28.039523 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.039530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.039736 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.039903 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:28.040374 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.041416 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.042909 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.043717 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.043843 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.044034 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.044124 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.044223 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":92,"threads_started":1}
I20260924 10:48:28.044368 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.038750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:39175 (local address 127.0.0.1:54986)
0924 10:48:28.039182 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.039223 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.039326 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.040078 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.040086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.040114 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:28.040255 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.040261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.041526 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.041531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.042554 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.042563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.043833 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.043837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.043942 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.043977 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.044084 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.044209 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260924 10:48:28.045135 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.038833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:36357 (local address 127.0.0.1:36918)
0924 10:48:28.040054 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.040066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.040167 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.040673 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.040676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.040691 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:28.040812 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.040821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.042243 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.042246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.043485 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.043495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.044607 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.044611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.044762 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.044814 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.044885 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.045010 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"mutex_wait_us":74,"thread_start_us":97,"threads_started":1}
I20260924 10:48:28.045292 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.039579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.2.182.1:39159)
0924 10:48:28.039981 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.039992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.040413 (+ 421us) server_negotiation.cc:407] Connection header received
0924 10:48:28.040529 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.040540 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.040735 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.040925 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:28.041111 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.042204 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.043478 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.044279 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.044386 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.044545 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.045042 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.045125 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":122,"threads_started":1}
I20260924 10:48:28.045437 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.039518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.2.182.3:36357)
0924 10:48:28.039894 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.039906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.040255 (+ 349us) server_negotiation.cc:407] Connection header received
0924 10:48:28.040337 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.040342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.040527 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.040684 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:28.040962 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.042109 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.043608 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.044310 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.044405 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.044545 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.044954 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.045051 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260924 10:48:28.045779 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.039428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39159 (local address 127.0.0.1:38102)
0924 10:48:28.040334 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.040347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.040429 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.040873 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.040876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.040925 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:28.041018 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.041024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.042286 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.042290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.043356 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.043365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.044607 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.044611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.044755 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.044812 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.044882 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.045430 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":509,"thread_start_us":77,"threads_started":1}
I20260924 10:48:28.047824 22760 tablet_service.cc:1467] Processing CreateTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 (DEFAULT_TABLE table=test-table [id=1bc83c2f7c654da09468437dbba2a488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:28.048075 22629 tablet_service.cc:1467] Processing CreateTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 (DEFAULT_TABLE table=test-table [id=1bc83c2f7c654da09468437dbba2a488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:28.048426 22891 tablet_service.cc:1467] Processing CreateTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 (DEFAULT_TABLE table=test-table [id=1bc83c2f7c654da09468437dbba2a488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:48:28.048820 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13019d28b2b9441f9daebfa84ebd4954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:28.048849 22760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13019d28b2b9441f9daebfa84ebd4954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:28.049252 22891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13019d28b2b9441f9daebfa84ebd4954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:28.054718 22974 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Bootstrap starting.
I20260924 10:48:28.055568 22975 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Bootstrap starting.
I20260924 10:48:28.056658 22976 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Bootstrap starting.
I20260924 10:48:28.057452 22974 tablet_bootstrap.cc:654] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:28.057538 22975 tablet_bootstrap.cc:654] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:28.058385 22975 log.cc:824] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:28.058395 22974 log.cc:824] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:28.058440 22976 tablet_bootstrap.cc:654] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:28.059226 22976 log.cc:824] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:28.060504 22974 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: No bootstrap required, opened a new log
I20260924 10:48:28.060504 22975 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: No bootstrap required, opened a new log
I20260924 10:48:28.060698 22974 ts_tablet_manager.cc:1397] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:48:28.060698 22975 ts_tablet_manager.cc:1397] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:48:28.060987 22976 tablet_bootstrap.cc:492] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: No bootstrap required, opened a new log
I20260924 10:48:28.061172 22976 ts_tablet_manager.cc:1397] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:48:28.064426 22976 raft_consensus.cc:348] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.064620 22976 raft_consensus.cc:374] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:28.064694 22976 raft_consensus.cc:729] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80b294e90294817b664b99b1c95d283, State: Initialized, Role: FOLLOWER
I20260924 10:48:28.064985 22974 raft_consensus.cc:348] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.064985 22975 raft_consensus.cc:348] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.065119 22976 consensus_queue.cc:260] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.065218 22974 raft_consensus.cc:374] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:28.065261 22975 raft_consensus.cc:374] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:28.065276 22974 raft_consensus.cc:729] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c30db5d936143a9b8a6fdda01128314, State: Initialized, Role: FOLLOWER
I20260924 10:48:28.065348 22975 raft_consensus.cc:729] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918eea415fa54354bdb2355f74d7cfd0, State: Initialized, Role: FOLLOWER
I20260924 10:48:28.065680 22957 heartbeater.cc:503] Master 127.2.182.62:46233 was elected leader, sending a full tablet report...
I20260924 10:48:28.065693 22974 consensus_queue.cc:260] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.065863 22975 consensus_queue.cc:260] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.066020 22976 ts_tablet_manager.cc:1428] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:48:28.066612 22974 ts_tablet_manager.cc:1428] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260924 10:48:28.066923 22695 heartbeater.cc:503] Master 127.2.182.62:46233 was elected leader, sending a full tablet report...
I20260924 10:48:28.066788 22975 ts_tablet_manager.cc:1428] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:48:28.067886 22826 heartbeater.cc:503] Master 127.2.182.62:46233 was elected leader, sending a full tablet report...
W20260924 10:48:28.087319 22827 tablet.cc:2369] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:48:28.166221 22696 tablet.cc:2369] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:48:28.250954 22958 tablet.cc:2369] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:28.260763 22982 raft_consensus.cc:482] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:48:28.261027 22982 raft_consensus.cc:504] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.262125 22982 leader_election.cc:288] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b80b294e90294817b664b99b1c95d283 (127.2.182.3:36357), 7c30db5d936143a9b8a6fdda01128314 (127.2.182.1:39159)
I20260924 10:48:28.265856 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.262423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.3:36357 (local address 127.2.182.2:43835)
0924 10:48:28.262586 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.262604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.262727 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.263023 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.263026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.263042 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:28.263158 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.263164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.263906 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.263910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.264696 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.264704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.265421 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.265424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.265544 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.265562 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.265647 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.265720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:48:28.265856 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.262633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:43835 (local address 127.2.182.3:36357)
0924 10:48:28.262772 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.262778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.262796 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:48:28.262861 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.262863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.262899 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.263013 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:28.263269 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.263789 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.264821 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.265296 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.265416 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.265476 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.265675 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.265743 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:28.266321 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.262533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.2:46307 (local address 127.2.182.1:39159)
0924 10:48:28.262677 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.262683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.263155 (+ 472us) server_negotiation.cc:407] Connection header received
0924 10:48:28.263314 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.263322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.263373 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.263454 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:48:28.263737 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.264216 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.265294 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.265702 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.265825 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.265881 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.266113 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.266185 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:48:28.266528 22911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13019d28b2b9441f9daebfa84ebd4954" candidate_uuid: "918eea415fa54354bdb2355f74d7cfd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80b294e90294817b664b99b1c95d283" is_pre_election: true
I20260924 10:48:28.266667 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.262423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39159 (local address 127.2.182.2:46307)
0924 10:48:28.263045 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.263058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:28.263187 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:28.263478 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:28.263481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:28.263496 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:48:28.263617 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.263622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.264373 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.264379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.265163 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:28.265172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.265837 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.265841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:28.265959 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.265987 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:48:28.266481 (+ 494us) client_negotiation.cc:240] Negotiation successful
0924 10:48:28.266560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":260,"thread_start_us":145,"threads_started":1}
I20260924 10:48:28.266878 22911 raft_consensus.cc:2454] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918eea415fa54354bdb2355f74d7cfd0 in term 0.
I20260924 10:48:28.267351 22649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13019d28b2b9441f9daebfa84ebd4954" candidate_uuid: "918eea415fa54354bdb2355f74d7cfd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c30db5d936143a9b8a6fdda01128314" is_pre_election: true
I20260924 10:48:28.267434 22713 leader_election.cc:302] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918eea415fa54354bdb2355f74d7cfd0, b80b294e90294817b664b99b1c95d283; no voters:
I20260924 10:48:28.267633 22649 raft_consensus.cc:2454] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918eea415fa54354bdb2355f74d7cfd0 in term 0.
I20260924 10:48:28.267738 22982 raft_consensus.cc:2781] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:48:28.267827 22982 raft_consensus.cc:482] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:48:28.267882 22982 raft_consensus.cc:3037] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:28.269022 22982 raft_consensus.cc:504] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.269341 22982 leader_election.cc:288] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [CANDIDATE]: Term 1 election: Requested vote from peers b80b294e90294817b664b99b1c95d283 (127.2.182.3:36357), 7c30db5d936143a9b8a6fdda01128314 (127.2.182.1:39159)
I20260924 10:48:28.269668 22911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13019d28b2b9441f9daebfa84ebd4954" candidate_uuid: "918eea415fa54354bdb2355f74d7cfd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80b294e90294817b664b99b1c95d283"
I20260924 10:48:28.269668 22649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13019d28b2b9441f9daebfa84ebd4954" candidate_uuid: "918eea415fa54354bdb2355f74d7cfd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c30db5d936143a9b8a6fdda01128314"
I20260924 10:48:28.269789 22911 raft_consensus.cc:3037] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:28.269789 22649 raft_consensus.cc:3037] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:28.271035 22649 raft_consensus.cc:2454] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918eea415fa54354bdb2355f74d7cfd0 in term 1.
I20260924 10:48:28.271035 22911 raft_consensus.cc:2454] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918eea415fa54354bdb2355f74d7cfd0 in term 1.
I20260924 10:48:28.271338 22713 leader_election.cc:302] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918eea415fa54354bdb2355f74d7cfd0, b80b294e90294817b664b99b1c95d283; no voters:
I20260924 10:48:28.271530 22982 raft_consensus.cc:2781] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:28.271761 22982 raft_consensus.cc:686] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 918eea415fa54354bdb2355f74d7cfd0, State: Running, Role: LEADER
I20260924 10:48:28.272135 22982 consensus_queue.cc:237] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } }
I20260924 10:48:28.275368 22505 catalog_manager.cc:5495] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 918eea415fa54354bdb2355f74d7cfd0 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "918eea415fa54354bdb2355f74d7cfd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918eea415fa54354bdb2355f74d7cfd0" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 39175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 } health_report { overall_health: UNKNOWN } } }
I20260924 10:48:28.336808 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.333709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.2.182.2:39175)
0924 10:48:28.333834 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:28.333838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:28.333852 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:48:28.333991 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:28.333994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:28.334048 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:28.334134 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:28.334332 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.334819 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.335606 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:28.335804 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:28.335877 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:28.336117 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:28.336519 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:28.336605 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:28.336660 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 10:48:28.336702 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:48:28.342765 22911 raft_consensus.cc:1261] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 1 FOLLOWER]: Refusing update from remote peer 918eea415fa54354bdb2355f74d7cfd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:28.342767 22649 raft_consensus.cc:1261] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 1 FOLLOWER]: Refusing update from remote peer 918eea415fa54354bdb2355f74d7cfd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:28.343499 22982 consensus_queue.cc:1035] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c30db5d936143a9b8a6fdda01128314" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 39159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:28.343693 22984 consensus_queue.cc:1035] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80b294e90294817b664b99b1c95d283" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 36357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:28.349982 22987 mvcc.cc:205] Tried to move back new op lower bound from 7332851336565092352 to 7332851336285331456. Current Snapshot: MvccSnapshot[applied={T|T < 7332851336565092352}]
I20260924 10:48:28.350669 22988 mvcc.cc:205] Tried to move back new op lower bound from 7332851336565092352 to 7332851336285331456. Current Snapshot: MvccSnapshot[applied={T|T < 7332851336565092352}]
I20260924 10:48:28.354213 22990 mvcc.cc:205] Tried to move back new op lower bound from 7332851336565092352 to 7332851336285331456. Current Snapshot: MvccSnapshot[applied={T|T < 7332851336565092352}]
I20260924 10:48:28.360522 22505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:28.360709 22505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:48:28.363209 22505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913827fda1374ab4b801810c4d95b93d: Sending DeleteTablet for 3 replicas of tablet 13019d28b2b9441f9daebfa84ebd4954
I20260924 10:48:28.364025 22629 tablet_service.cc:1514] Processing DeleteTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:28 UTC) from {username='slave'} at 127.0.0.1:38102
I20260924 10:48:28.364277 22891 tablet_service.cc:1514] Processing DeleteTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:28 UTC) from {username='slave'} at 127.0.0.1:36918
I20260924 10:48:28.364508 22997 tablet_replica.cc:330] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: stopping tablet replica
I20260924 10:48:28.364698 22998 tablet_replica.cc:330] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: stopping tablet replica
I20260924 10:48:28.364818 22997 raft_consensus.cc:2229] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:28.364847 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22567
I20260924 10:48:28.365015 22998 raft_consensus.cc:2229] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:48:28.365149 22997 raft_consensus.cc:2258] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:28.365280 22998 raft_consensus.cc:2258] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:28.366659 22997 ts_tablet_manager.cc:1905] T 13019d28b2b9441f9daebfa84ebd4954 P 7c30db5d936143a9b8a6fdda01128314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:28.366729 22998 ts_tablet_manager.cc:1905] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:48:28.369720 22760 tablet_service.cc:1514] Processing DeleteTablet for tablet 13019d28b2b9441f9daebfa84ebd4954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:48:28 UTC) from {username='slave'} at 127.0.0.1:54986
I20260924 10:48:28.370126 22998 ts_tablet_manager.cc:1918] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:48:28.370230 22998 log.cc:1196] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Deleting WAL directory at /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246662595599-2776-0/minicluster-data/ts-2/wal/wals/13019d28b2b9441f9daebfa84ebd4954
I20260924 10:48:28.370386 22999 tablet_replica.cc:330] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: stopping tablet replica
I20260924 10:48:28.370554 22998 ts_tablet_manager.cc:1939] T 13019d28b2b9441f9daebfa84ebd4954 P b80b294e90294817b664b99b1c95d283: Deleting consensus metadata
I20260924 10:48:28.370879 22999 raft_consensus.cc:2229] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:48:28.371361 22999 raft_consensus.cc:2258] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:48:28.371829 22491 catalog_manager.cc:4841] TS b80b294e90294817b664b99b1c95d283 (127.2.182.3:36357): tablet 13019d28b2b9441f9daebfa84ebd4954 (table test-table [id=1bc83c2f7c654da09468437dbba2a488]) successfully deleted
I20260924 10:48:28.373099 22999 ts_tablet_manager.cc:1905] T 13019d28b2b9441f9daebfa84ebd4954 P 918eea415fa54354bdb2355f74d7cfd0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:48:28.374697 22493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.1:39159 (error 108)
I20260924 10:48:28.375731 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.375416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.1:39159 (local address 127.0.0.1:38114)
0924 10:48:28.375549 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.375626 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.1:39159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:48:28.375824 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22698
W20260924 10:48:28.376421 22493 catalog_manager.cc:4584] TS 7c30db5d936143a9b8a6fdda01128314 (127.2.182.1:39159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13019d28b2b9441f9daebfa84ebd4954: Network error: Client connection negotiation failed: client connection to 127.2.182.1:39159: connect: Connection refused (error 111)
I20260924 10:48:28.381027 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22829
I20260924 10:48:28.382174 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:28.381308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.2:39175 (local address 127.0.0.1:55002)
0924 10:48:28.381990 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:28.382055 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.2:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572}
I20260924 10:48:28.386567 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 22476
2026-09-24T10:48:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1170 ms)
[----------] 6 tests from EncryptionPolicyTest (10066 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:28 dist-test-slave-6lwt krb5kdc[23003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:28 dist-test-slave-6lwt krb5kdc[23003](info): set up 2 sockets
Sep 24 10:48:28 dist-test-slave-6lwt krb5kdc[23003](info): commencing operation
krb5kdc: starting...
W20260924 10:48:29.437810 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:29.559427 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:29.663477 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:29.663733 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:29.663802 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:29.667722 23016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:29.667805 23016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:29.667843 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:29.667899 23016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:29.667938 23016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:29.672830 23016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:29.673815 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:29.674759 23016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:29.682826 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:29.682832 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:29.682832 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:29.682864 23016 server_base.cc:1034] running on GCE node
I20260924 10:48:29.683616 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:29.684609 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:29.685786 23016 hybrid_clock.cc:648] HybridClock initialized: now 1790246909685750 us; error 53 us; skew 500 ppm
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:29.688311 23016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:29.689182 23016 webserver.cc:460] Webserver started at http://127.0.0.1:37877/ using document root <none> and password file <none>
I20260924 10:48:29.689720 23016 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:29.689808 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:29.690025 23016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:29.691896 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6"
format_stamp: "Formatted at 2026-09-24 10:48:29 on dist-test-slave-6lwt"
server_key: "2013ef5bb1f7217b2e3d651ca21512d3"
server_key_iv: "f3916ad70ff0f3191d3372d73904a291"
server_key_version: "encryptionkey@0"
I20260924 10:48:29.692487 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6"
format_stamp: "Formatted at 2026-09-24 10:48:29 on dist-test-slave-6lwt"
server_key: "2013ef5bb1f7217b2e3d651ca21512d3"
server_key_iv: "f3916ad70ff0f3191d3372d73904a291"
server_key_version: "encryptionkey@0"
I20260924 10:48:29.695911 23016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:29.698189 23031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:29.699270 23016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:29.699401 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6"
format_stamp: "Formatted at 2026-09-24 10:48:29 on dist-test-slave-6lwt"
server_key: "2013ef5bb1f7217b2e3d651ca21512d3"
server_key_iv: "f3916ad70ff0f3191d3372d73904a291"
server_key_version: "encryptionkey@0"
I20260924 10:48:29.699504 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:29.715507 23016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:29.718252 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:29.718451 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:29.725312 23016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44077
I20260924 10:48:29.725333 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44077 every 8 connection(s)
I20260924 10:48:29.726193 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:29.728955 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:29.734655 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23016
I20260924 10:48:29.734789 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:29.735018 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: Bootstrap starting.
I20260924 10:48:29.735096 2776 external_mini_cluster.cc:1428] Setting key 0a39c5719bdd0b5104174f36883f38f9
I20260924 10:48:29.737469 23084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:29.738186 23084 log.cc:824] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:29.740108 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: No bootstrap required, opened a new log
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:48:29.744489 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260924 10:48:29.744679 23084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:29.744735 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2d9dbe9fe04837801c4dd8d3cf40c6, State: Initialized, Role: FOLLOWER
I20260924 10:48:29.745282 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260924 10:48:29.745415 23084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:29.745522 23084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:29.745640 23084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:29.746412 23084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260924 10:48:29.746740 23084 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2d9dbe9fe04837801c4dd8d3cf40c6; no voters:
I20260924 10:48:29.747023 23084 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:29.747136 23089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:29.747524 23089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2d9dbe9fe04837801c4dd8d3cf40c6, State: Running, Role: LEADER
I20260924 10:48:29.747889 23089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260924 10:48:29.748242 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:29.749171 23089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2d9dbe9fe04837801c4dd8d3cf40c6. Latest consensus state: current_term: 1 leader_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } } }
I20260924 10:48:29.749961 23089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:29.749835 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d9dbe9fe04837801c4dd8d3cf40c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } } }
I20260924 10:48:29.751266 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:29.736438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55097 (local address 127.0.0.1:44077)
0924 10:48:29.736873 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:29.736940 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:29.737010 (+ 70us) server_negotiation.cc:407] Connection header received
0924 10:48:29.737730 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:29.737751 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:29.738071 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:29.738419 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:29.739210 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:29.740213 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:29.741375 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:29.741836 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:29.741962 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:29.744971 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:29.744977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:29.744989 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:29.745015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:29.747103 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:29.747683 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:29.747689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:29.747694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:29.747843 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:29.748225 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:29.748229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:29.748230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:29.748760 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:29.749028 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:29.750058 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0924 10:48:29.750291 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":123,"threads_started":1}
I20260924 10:48:29.751518 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6 [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:29.752990 23104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:29.753114 23104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:29.753217 23103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:29.754114 23103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:29.754751 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:29.760003 23103 catalog_manager.cc:1347] Generated new cluster ID: 07e217e1c09a4af3b3fbe59ec1a659f2
I20260924 10:48:29.760059 23103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:29.765547 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:29.757027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56109 (local address 127.0.0.1:44077)
0924 10:48:29.757133 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:29.757137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:29.757393 (+ 256us) server_negotiation.cc:407] Connection header received
0924 10:48:29.757509 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:29.757512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:29.757555 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:29.757639 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:29.758488 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:29.758991 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:29.759854 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:29.760200 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:29.760309 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:29.762055 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:29.762058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:29.762061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:29.762085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:29.763706 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:29.764271 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:29.764274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:29.764276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:29.764354 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:29.764709 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:29.764712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:29.764713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:29.764921 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:29.765059 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:29.765328 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:48:29.765412 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:29.768651 23103 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:29.769493 23103 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:29.777278 23103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2d9dbe9fe04837801c4dd8d3cf40c6: Generated new TSK 0
I20260924 10:48:29.777961 23103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:29.785430 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23016
W20260924 10:48:29.790796 23001 connection.cc:535] client connection to 127.0.0.1:44077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44077): Connection reset by peer (error 104)
2026-09-24T10:48:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23115](info): set up 2 sockets
Sep 24 10:48:29 dist-test-slave-6lwt krb5kdc[23115](info): commencing operation
krb5kdc: starting...
W20260924 10:48:30.789040 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:30 dist-test-slave-6lwt krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:30.906123 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:31.011165 23128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:31.011490 23128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:31.011581 23128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:31.015569 23128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:31.015667 23128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:31.015705 23128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:31.015741 23128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:31.015770 23128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:31.020859 23128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:31.021881 23128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:31.022876 23128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:31.030989 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:31.031036 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:31.030992 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:31.031301 23128 server_base.cc:1034] running on GCE node
I20260924 10:48:31.031822 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:31.032783 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:31.033933 23128 hybrid_clock.cc:648] HybridClock initialized: now 1790246911033907 us; error 42 us; skew 500 ppm
Sep 24 10:48:31 dist-test-slave-6lwt krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:31.036545 23128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:31.037412 23128 webserver.cc:460] Webserver started at http://127.0.0.1:40479/ using document root <none> and password file <none>
I20260924 10:48:31.037914 23128 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:31.037971 23128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:31.038137 23128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:31.040002 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "80eb9a7ac3364d26908077e20ccdc811"
format_stamp: "Formatted at 2026-09-24 10:48:31 on dist-test-slave-6lwt"
server_key: "e8157719d2388dadc089a32888bbcf4b"
server_key_iv: "e14875db1612bdc39f1936e752f16691"
server_key_version: "encryptionkey@0"
I20260924 10:48:31.040478 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "80eb9a7ac3364d26908077e20ccdc811"
format_stamp: "Formatted at 2026-09-24 10:48:31 on dist-test-slave-6lwt"
server_key: "e8157719d2388dadc089a32888bbcf4b"
server_key_iv: "e14875db1612bdc39f1936e752f16691"
server_key_version: "encryptionkey@0"
I20260924 10:48:31.043953 23128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:48:31.046135 23143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:31.047245 23128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:31.047370 23128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "80eb9a7ac3364d26908077e20ccdc811"
format_stamp: "Formatted at 2026-09-24 10:48:31 on dist-test-slave-6lwt"
server_key: "e8157719d2388dadc089a32888bbcf4b"
server_key_iv: "e14875db1612bdc39f1936e752f16691"
server_key_version: "encryptionkey@0"
I20260924 10:48:31.047465 23128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:31.073264 23128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:31.076468 23128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:31.076615 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:31.083493 23128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43171
I20260924 10:48:31.083519 23195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43171 every 8 connection(s)
I20260924 10:48:31.084383 23128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:31.086895 23196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:31.091975 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811: Bootstrap starting.
I20260924 10:48:31.092054 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23128
I20260924 10:48:31.092193 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:31.092490 2776 external_mini_cluster.cc:1428] Setting key c23f5d33f812a787eaa38902a291e561
I20260924 10:48:31.094409 23196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:31.095103 23196 log.cc:824] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:31.096807 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811: No bootstrap required, opened a new log
I20260924 10:48:31.100332 23196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260924 10:48:31.100594 23196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:31.100687 23196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80eb9a7ac3364d26908077e20ccdc811, State: Initialized, Role: FOLLOWER
I20260924 10:48:31.101140 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:31.093754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48775 (local address 127.0.0.1:43171)
0924 10:48:31.094162 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:31.094175 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:31.094207 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:48:31.094850 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:31.094867 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:31.095183 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:31.095538 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:31.095951 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:31.096932 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:31.097822 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:31.098283 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:31.098446 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:31.099414 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:31.099419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:31.099430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:31.099789 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:31.099837 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:31.099858 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:31.099990 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:31.100172 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 10:48:31.100305 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260924 10:48:31.101166 23196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260924 10:48:31.101344 23196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:31.101429 23196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:31.101531 23196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:31.102317 23196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260924 10:48:31.102648 23196 leader_election.cc:302] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80eb9a7ac3364d26908077e20ccdc811; no voters:
I20260924 10:48:31.102950 23196 leader_election.cc:288] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:31.103314 23201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:31.103556 23201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [term 1 LEADER]: Becoming Leader. State: Replica: 80eb9a7ac3364d26908077e20ccdc811, State: Running, Role: LEADER
I20260924 10:48:31.103917 23201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260924 10:48:31.103982 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:31.104206 23196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:31.105497 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80eb9a7ac3364d26908077e20ccdc811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } } }
I20260924 10:48:31.105494 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80eb9a7ac3364d26908077e20ccdc811. Latest consensus state: current_term: 1 leader_uuid: "80eb9a7ac3364d26908077e20ccdc811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80eb9a7ac3364d26908077e20ccdc811" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } } }
I20260924 10:48:31.105625 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:31.105625 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:31.106114 23216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:31.109201 23216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:31.110988 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:31.106920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45105 (local address 127.0.0.1:43171)
0924 10:48:31.107019 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:31.107023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:31.107213 (+ 190us) server_negotiation.cc:407] Connection header received
0924 10:48:31.107383 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:31.107385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:31.107426 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:31.107474 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:31.107875 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:31.108400 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:31.109307 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:31.109552 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:31.109662 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:31.110494 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:31.110497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:31.110500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:31.110512 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:31.110542 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:31.110552 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:31.110656 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:31.110798 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:48:31.110829 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:48:31.114742 23216 catalog_manager.cc:1347] Generated new cluster ID: 9d0e40e865a043d1810996cf2ffde99d
I20260924 10:48:31.114800 23216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:31.144776 23216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:31.145680 23216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:31.162541 23216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80eb9a7ac3364d26908077e20ccdc811: Generated new TSK 0
I20260924 10:48:31.163237 23216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:31.175542 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23128
W20260924 10:48:31.180814 23113 connection.cc:535] client connection to 127.0.0.1:43171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43171): Connection reset by peer (error 104)
2026-09-24T10:48:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:31 dist-test-slave-6lwt krb5kdc[23227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:31 dist-test-slave-6lwt krb5kdc[23227](info): set up 2 sockets
Sep 24 10:48:31 dist-test-slave-6lwt krb5kdc[23227](info): commencing operation
krb5kdc: starting...
W20260924 10:48:32.217342 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:32 dist-test-slave-6lwt krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:32.336997 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:32.442559 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:32.442794 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:32.442833 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:32.446866 23240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:32.446918 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:32.446964 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:32.447001 23240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:32.447037 23240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:32.452073 23240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:32.453061 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:32.454033 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:32.462153 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:32.462175 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:32.462266 23240 server_base.cc:1034] running on GCE node
W20260924 10:48:32.462153 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:32.462950 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:32.463929 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:32.465111 23240 hybrid_clock.cc:648] HybridClock initialized: now 1790246912465084 us; error 40 us; skew 500 ppm
Sep 24 10:48:32 dist-test-slave-6lwt krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:32.467842 23240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:32.468739 23240 webserver.cc:460] Webserver started at http://127.0.0.1:35493/ using document root <none> and password file <none>
I20260924 10:48:32.469331 23240 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:32.469477 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:32.469715 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:32.471629 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "4c7204410fe248d792a9c2e67640679d"
format_stamp: "Formatted at 2026-09-24 10:48:32 on dist-test-slave-6lwt"
server_key: "25c64183d500874bce0291fcef2d4300"
server_key_iv: "4e255703359a5d348af092f2401467c7"
server_key_version: "encryptionkey@0"
I20260924 10:48:32.472184 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "4c7204410fe248d792a9c2e67640679d"
format_stamp: "Formatted at 2026-09-24 10:48:32 on dist-test-slave-6lwt"
server_key: "25c64183d500874bce0291fcef2d4300"
server_key_iv: "4e255703359a5d348af092f2401467c7"
server_key_version: "encryptionkey@0"
I20260924 10:48:32.475704 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:32.478214 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:32.479465 23240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:32.479614 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4c7204410fe248d792a9c2e67640679d"
format_stamp: "Formatted at 2026-09-24 10:48:32 on dist-test-slave-6lwt"
server_key: "25c64183d500874bce0291fcef2d4300"
server_key_iv: "4e255703359a5d348af092f2401467c7"
server_key_version: "encryptionkey@0"
I20260924 10:48:32.479725 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:32.484575 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:32.487795 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:32.487991 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:32.494823 23240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33957
I20260924 10:48:32.494836 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33957 every 8 connection(s)
I20260924 10:48:32.495712 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:32.498386 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:32.502411 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23240
I20260924 10:48:32.502549 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:32.502847 2776 external_mini_cluster.cc:1428] Setting key 0fec6ba9ff2aad61e428bbd6c507692a
I20260924 10:48:32.503997 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d: Bootstrap starting.
I20260924 10:48:32.506237 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:32.506981 23308 log.cc:824] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:32.507927 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:32.504136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36237 (local address 127.0.0.1:33957)
0924 10:48:32.504544 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:32.504558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:32.504592 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:48:32.505257 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:32.505275 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:32.505550 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:32.505863 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:32.506625 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:32.506630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:32.506640 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:32.506650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:32.506701 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:32.506719 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:32.506825 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:32.506950 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:48:32.507120 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260924 10:48:32.508854 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d: No bootstrap required, opened a new log
I20260924 10:48:32.510643 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:32.512790 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260924 10:48:32.513120 23308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:32.513183 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7204410fe248d792a9c2e67640679d, State: Initialized, Role: FOLLOWER
I20260924 10:48:32.513644 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260924 10:48:32.513833 23308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:32.513917 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:32.514017 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:32.514133 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:32.512574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53025 (local address 127.0.0.1:33957)
0924 10:48:32.512692 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:32.512696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:32.512861 (+ 165us) server_negotiation.cc:407] Connection header received
0924 10:48:32.513003 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:32.513005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:32.513031 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:32.513117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:32.513735 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:32.513737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:32.513739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:32.513749 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:32.513791 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:32.513806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:32.513895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:32.513992 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:48:32.514027 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:48:32.515102 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260924 10:48:32.515427 23308 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c7204410fe248d792a9c2e67640679d; no voters:
I20260924 10:48:32.515720 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:32.515802 23318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:32.516019 23318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [term 1 LEADER]: Becoming Leader. State: Replica: 4c7204410fe248d792a9c2e67640679d, State: Running, Role: LEADER
I20260924 10:48:32.516335 23318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260924 10:48:32.516880 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:32.517903 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c7204410fe248d792a9c2e67640679d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260924 10:48:32.518018 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:32.518151 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7204410fe248d792a9c2e67640679d. Latest consensus state: current_term: 1 leader_uuid: "4c7204410fe248d792a9c2e67640679d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7204410fe248d792a9c2e67640679d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260924 10:48:32.518301 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:32.518606 23327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:32.521418 23327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:32.526219 23327 catalog_manager.cc:1347] Generated new cluster ID: 732ea746337a4a06be6a2ea7a4a84680
I20260924 10:48:32.526286 23327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:32.555383 23327 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:32.555724 23327 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:32.560478 23327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7204410fe248d792a9c2e67640679d: Generated new TSK 0
I20260924 10:48:32.561159 23327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:32.568076 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23240
2026-09-24T10:48:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:32 dist-test-slave-6lwt krb5kdc[23339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:32 dist-test-slave-6lwt krb5kdc[23339](info): set up 2 sockets
Sep 24 10:48:32 dist-test-slave-6lwt krb5kdc[23339](info): commencing operation
krb5kdc: starting...
W20260924 10:48:33.573400 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:33.688472 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:33.793382 23352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:33.793641 23352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:33.793713 23352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:33.797485 23352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:33.797562 23352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:33.797600 23352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:33.797655 23352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:33.797693 23352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:33.802609 23352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:33.803589 23352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:33.804534 23352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:33.812300 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:33.812408 23352 server_base.cc:1034] running on GCE node
W20260924 10:48:33.812507 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:33.812438 23357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:33.813138 23352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:33.814092 23352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:33.815271 23352 hybrid_clock.cc:648] HybridClock initialized: now 1790246913815259 us; error 23 us; skew 500 ppm
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:33.824411 23352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:33.825320 23352 webserver.cc:460] Webserver started at http://127.0.0.1:41211/ using document root <none> and password file <none>
I20260924 10:48:33.825835 23352 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:33.825932 23352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:33.826148 23352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:33.828028 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "edf037c3801f488a940080f77c1de38e"
format_stamp: "Formatted at 2026-09-24 10:48:33 on dist-test-slave-6lwt"
server_key: "32b54ffa25ae77c858aa1dba267a5180"
server_key_iv: "c57918a16786ab2fb7218411f6e1d157"
server_key_version: "encryptionkey@0"
I20260924 10:48:33.828554 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "edf037c3801f488a940080f77c1de38e"
format_stamp: "Formatted at 2026-09-24 10:48:33 on dist-test-slave-6lwt"
server_key: "32b54ffa25ae77c858aa1dba267a5180"
server_key_iv: "c57918a16786ab2fb7218411f6e1d157"
server_key_version: "encryptionkey@0"
I20260924 10:48:33.831979 23352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:33.834192 23367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:33.835239 23352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:33.835371 23352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "edf037c3801f488a940080f77c1de38e"
format_stamp: "Formatted at 2026-09-24 10:48:33 on dist-test-slave-6lwt"
server_key: "32b54ffa25ae77c858aa1dba267a5180"
server_key_iv: "c57918a16786ab2fb7218411f6e1d157"
server_key_version: "encryptionkey@0"
I20260924 10:48:33.835474 23352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:33.853340 23352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:33.856350 23352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:33.856540 23352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:33.862876 23352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35877
I20260924 10:48:33.862888 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35877 every 8 connection(s)
I20260924 10:48:33.863747 23352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:33.866336 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:33.871428 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e: Bootstrap starting.
I20260924 10:48:33.873813 23420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:33.873842 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23352
I20260924 10:48:33.874017 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:33.874287 2776 external_mini_cluster.cc:1428] Setting key 189f65d00f845de2728037900c507baa
I20260924 10:48:33.874588 23420 log.cc:824] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:33.876677 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e: No bootstrap required, opened a new log
I20260924 10:48:33.880837 23420 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260924 10:48:33.881100 23420 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:33.881163 23420 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf037c3801f488a940080f77c1de38e, State: Initialized, Role: FOLLOWER
I20260924 10:48:33.881629 23420 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260924 10:48:33.881793 23420 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:33.881866 23420 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:33.881984 23420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:33.882797 23420 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260924 10:48:33.883132 23420 leader_election.cc:302] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf037c3801f488a940080f77c1de38e; no voters:
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:48:33.883414 23420 leader_election.cc:288] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:33.883517 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:33.883773 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [term 1 LEADER]: Becoming Leader. State: Replica: edf037c3801f488a940080f77c1de38e, State: Running, Role: LEADER
I20260924 10:48:33.884135 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260924 10:48:33.884788 23420 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:33.885890 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf037c3801f488a940080f77c1de38e. Latest consensus state: current_term: 1 leader_uuid: "edf037c3801f488a940080f77c1de38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260924 10:48:33.886009 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:33.885876 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf037c3801f488a940080f77c1de38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf037c3801f488a940080f77c1de38e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260924 10:48:33.886132 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:33.886415 23434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:33.889269 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:33.875301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45895 (local address 127.0.0.1:35877)
0924 10:48:33.875708 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:33.875725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:33.875764 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:48:33.876612 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:33.876633 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:33.877074 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:33.877411 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:33.878092 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:33.879598 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:33.880421 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:33.881131 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:33.881290 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:33.884044 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:33.884050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:33.884064 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:33.884092 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:33.886290 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:33.886805 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:33.886810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:33.886815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:33.886941 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:33.887245 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:33.887248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:33.887268 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:33.887745 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:33.887927 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:33.888205 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:48:33.888473 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260924 10:48:33.889514 23434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:33.891913 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:33.895183 23434 catalog_manager.cc:1347] Generated new cluster ID: adc676106a7644889f09c6608ba32610
I20260924 10:48:33.895251 23434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:33.904129 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:33.893856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55637 (local address 127.0.0.1:35877)
0924 10:48:33.893990 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:33.893994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:33.895497 (+ 1503us) server_negotiation.cc:407] Connection header received
0924 10:48:33.895659 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:33.895662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:33.895702 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:33.895749 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:33.896590 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:33.897389 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:33.898518 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:33.898877 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:33.898954 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:33.900604 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:33.900609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:33.900612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:33.900653 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:33.902362 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:33.902815 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:33.902820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:33.902822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:33.902910 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:33.903238 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:33.903241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:33.903243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:33.903435 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:33.903530 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:33.903876 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 10:48:33.903982 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:33.907373 23434 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:33.908447 23434 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:33.915145 23434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf037c3801f488a940080f77c1de38e: Generated new TSK 0
I20260924 10:48:33.915791 23434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:33.923988 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23352
2026-09-24T10:48:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23451](info): set up 2 sockets
Sep 24 10:48:33 dist-test-slave-6lwt krb5kdc[23451](info): commencing operation
krb5kdc: starting...
W20260924 10:48:34.933010 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:35 dist-test-slave-6lwt krb5kdc[23451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:35.051301 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:35.163179 23464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:35.163434 23464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:35.163506 23464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:35.167557 23464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:35.167625 23464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:35.167650 23464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:35.167670 23464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:35.167737 23464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:35.173168 23464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:35.174273 23464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:35.175388 23464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:35.184058 23469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:35.184058 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:35.184222 23464 server_base.cc:1034] running on GCE node
W20260924 10:48:35.184396 23470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:35.185062 23464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:35.186228 23464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:35.187426 23464 hybrid_clock.cc:648] HybridClock initialized: now 1790246915187410 us; error 34 us; skew 500 ppm
Sep 24 10:48:35 dist-test-slave-6lwt krb5kdc[23451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:35.190284 23464 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:35.191256 23464 webserver.cc:460] Webserver started at http://127.0.0.1:41731/ using document root <none> and password file <none>
I20260924 10:48:35.191821 23464 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:35.191917 23464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:35.192164 23464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:35.194303 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "30d4e20a7b344e13bf1bcfc2567c6b26"
format_stamp: "Formatted at 2026-09-24 10:48:35 on dist-test-slave-6lwt"
server_key: "74fde1423ea60837d107bbffb427ce4d"
server_key_iv: "cee520b442e1e756a2b100ca045936ed"
server_key_version: "encryptionkey@0"
I20260924 10:48:35.194890 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "30d4e20a7b344e13bf1bcfc2567c6b26"
format_stamp: "Formatted at 2026-09-24 10:48:35 on dist-test-slave-6lwt"
server_key: "74fde1423ea60837d107bbffb427ce4d"
server_key_iv: "cee520b442e1e756a2b100ca045936ed"
server_key_version: "encryptionkey@0"
I20260924 10:48:35.198675 23464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:48:35.201031 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:35.202173 23464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:35.202322 23464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "30d4e20a7b344e13bf1bcfc2567c6b26"
format_stamp: "Formatted at 2026-09-24 10:48:35 on dist-test-slave-6lwt"
server_key: "74fde1423ea60837d107bbffb427ce4d"
server_key_iv: "cee520b442e1e756a2b100ca045936ed"
server_key_version: "encryptionkey@0"
I20260924 10:48:35.202446 23464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:35.226706 23464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:35.229292 23464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:35.229489 23464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:35.236430 23464 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36753
I20260924 10:48:35.236450 23531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36753 every 8 connection(s)
I20260924 10:48:35.237435 23464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:35.239959 23532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:35.245255 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26: Bootstrap starting.
I20260924 10:48:35.246923 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23464
I20260924 10:48:35.247061 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:35.247325 2776 external_mini_cluster.cc:1428] Setting key 5ed7cb68148c221dfb2d91d59e0de467
I20260924 10:48:35.247802 23532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:35.248631 23532 log.cc:824] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:35.250589 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26: No bootstrap required, opened a new log
I20260924 10:48:35.254053 23532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } }
I20260924 10:48:35.254367 23532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:35.254449 23532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d4e20a7b344e13bf1bcfc2567c6b26, State: Initialized, Role: FOLLOWER
I20260924 10:48:35.254871 23532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } }
I20260924 10:48:35.255038 23532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:35.255119 23532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:35.255234 23532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:35.256230 23532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } }
I20260924 10:48:35.256559 23532 leader_election.cc:302] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d4e20a7b344e13bf1bcfc2567c6b26; no voters:
I20260924 10:48:35.256697 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:35.248420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35259 (local address 127.0.0.1:36753)
0924 10:48:35.248862 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:35.248877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:35.248969 (+ 92us) server_negotiation.cc:407] Connection header received
0924 10:48:35.249590 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:35.249606 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:35.249920 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:35.250183 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:35.250546 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:35.251538 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:35.252755 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:35.253779 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:35.253902 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:35.254879 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:35.254884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:35.254896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:35.255394 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:35.255455 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:35.255478 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:35.255581 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:35.255721 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 10:48:35.255928 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260924 10:48:35.256924 23532 leader_election.cc:288] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:35.257308 23538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:35.257581 23538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [term 1 LEADER]: Becoming Leader. State: Replica: 30d4e20a7b344e13bf1bcfc2567c6b26, State: Running, Role: LEADER
I20260924 10:48:35.257931 23538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } }
I20260924 10:48:35.258193 23532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:35.259713 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } } }
I20260924 10:48:35.259848 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:35.260164 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:35.260215 23548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:35.261173 23537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d4e20a7b344e13bf1bcfc2567c6b26. Latest consensus state: current_term: 1 leader_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4e20a7b344e13bf1bcfc2567c6b26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36753 } } }
I20260924 10:48:35.261273 23537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:35.263365 23548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:35.266578 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:35.262470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46575 (local address 127.0.0.1:36753)
0924 10:48:35.262578 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:35.262582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:35.262820 (+ 238us) server_negotiation.cc:407] Connection header received
0924 10:48:35.262964 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:35.262968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:35.263016 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:35.263100 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:35.263359 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:35.263911 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:35.264840 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:35.265136 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:35.265248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:35.266130 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:35.266134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:35.266137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:35.266148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:35.266181 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:35.266191 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:35.266266 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:35.266368 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 10:48:35.266421 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:48:35.268987 23548 catalog_manager.cc:1347] Generated new cluster ID: 5b587cec08924859bcfc861b5d216ab9
I20260924 10:48:35.269047 23548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:35.285574 23548 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:35.286458 23548 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:35.294751 23548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d4e20a7b344e13bf1bcfc2567c6b26: Generated new TSK 0
I20260924 10:48:35.295346 23548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:35.301716 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23464
2026-09-24T10:48:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:35 dist-test-slave-6lwt krb5kdc[23563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:35 dist-test-slave-6lwt krb5kdc[23563](info): set up 2 sockets
Sep 24 10:48:35 dist-test-slave-6lwt krb5kdc[23563](info): commencing operation
krb5kdc: starting...
W20260924 10:48:36.321377 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:36 dist-test-slave-6lwt krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:36.435331 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:36.544370 23576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:36.544646 23576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:36.544726 23576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:36.548559 23576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:36.548650 23576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:36.548687 23576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:36.548741 23576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:36.548781 23576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:36.553969 23576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:36.555030 23576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:36.556028 23576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:36.564356 23581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:36.564345 23584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:36.564482 23576 server_base.cc:1034] running on GCE node
W20260924 10:48:36.564343 23582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:36.565224 23576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:36.566231 23576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:36.567385 23576 hybrid_clock.cc:648] HybridClock initialized: now 1790246916567355 us; error 42 us; skew 500 ppm
Sep 24 10:48:36 dist-test-slave-6lwt krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:36.570225 23576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:36.571043 23576 webserver.cc:460] Webserver started at http://127.0.0.1:40199/ using document root <none> and password file <none>
I20260924 10:48:36.571596 23576 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:36.571656 23576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:36.571877 23576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:36.573911 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "50b9d080d5114657aeee808e86a64312"
format_stamp: "Formatted at 2026-09-24 10:48:36 on dist-test-slave-6lwt"
server_key: "158ec22b0e90839ab2699076b0377438"
server_key_iv: "30e325085d61ce92c07bb88d95e42940"
server_key_version: "encryptionkey@0"
I20260924 10:48:36.574465 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "50b9d080d5114657aeee808e86a64312"
format_stamp: "Formatted at 2026-09-24 10:48:36 on dist-test-slave-6lwt"
server_key: "158ec22b0e90839ab2699076b0377438"
server_key_iv: "30e325085d61ce92c07bb88d95e42940"
server_key_version: "encryptionkey@0"
I20260924 10:48:36.578074 23576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:36.580488 23591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:36.581717 23576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:36.581846 23576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "50b9d080d5114657aeee808e86a64312"
format_stamp: "Formatted at 2026-09-24 10:48:36 on dist-test-slave-6lwt"
server_key: "158ec22b0e90839ab2699076b0377438"
server_key_iv: "30e325085d61ce92c07bb88d95e42940"
server_key_version: "encryptionkey@0"
I20260924 10:48:36.581944 23576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:36.586365 23576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:36.589082 23576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:36.589258 23576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:36.595953 23576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35535
I20260924 10:48:36.595973 23643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35535 every 8 connection(s)
I20260924 10:48:36.596854 23576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:36.599452 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:36.600745 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23576
I20260924 10:48:36.600929 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:36.601245 2776 external_mini_cluster.cc:1428] Setting key 3fa4e80124baa9b09843ba5c9a1d5e12
I20260924 10:48:36.605715 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312: Bootstrap starting.
I20260924 10:48:36.606340 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:36.602626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54785 (local address 127.0.0.1:35535)
0924 10:48:36.603058 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:36.603073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:36.603107 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:48:36.603702 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:36.603717 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:36.603995 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:36.604320 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:36.605174 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:36.605179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:36.605190 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:36.605201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:36.605252 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:36.605271 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:36.605419 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:36.605489 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 10:48:36.605640 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260924 10:48:36.608119 23644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:36.609062 23644 log.cc:824] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:36.609099 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:36.610809 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312: No bootstrap required, opened a new log
I20260924 10:48:36.612572 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:36.611017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39625 (local address 127.0.0.1:35535)
0924 10:48:36.611148 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:36.611152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:36.611361 (+ 209us) server_negotiation.cc:407] Connection header received
0924 10:48:36.611472 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:36.611474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:36.611499 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:36.611561 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:36.612204 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:36.612206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:36.612208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:36.612217 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:36.612240 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:36.612249 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:36.612315 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:36.612415 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 10:48:36.612443 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:48:36.615144 23644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } }
I20260924 10:48:36.615434 23644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:36.615494 23644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b9d080d5114657aeee808e86a64312, State: Initialized, Role: FOLLOWER
I20260924 10:48:36.615923 23644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } }
I20260924 10:48:36.616070 23644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:36.616133 23644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:36.616225 23644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:36.617136 23644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } }
I20260924 10:48:36.617498 23644 leader_election.cc:302] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b9d080d5114657aeee808e86a64312; no voters:
I20260924 10:48:36.617851 23644 leader_election.cc:288] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:36.617949 23654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:36.618212 23654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [term 1 LEADER]: Becoming Leader. State: Replica: 50b9d080d5114657aeee808e86a64312, State: Running, Role: LEADER
I20260924 10:48:36.618630 23654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } }
I20260924 10:48:36.618991 23644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:36.620270 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b9d080d5114657aeee808e86a64312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } } }
I20260924 10:48:36.620276 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b9d080d5114657aeee808e86a64312. Latest consensus state: current_term: 1 leader_uuid: "50b9d080d5114657aeee808e86a64312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b9d080d5114657aeee808e86a64312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35535 } } }
I20260924 10:48:36.620410 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:36.620410 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:36.620858 23666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:36.623950 23666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:36.629463 23666 catalog_manager.cc:1347] Generated new cluster ID: 3b817fd59b7e47328b1b655fdc197518
I20260924 10:48:36.629534 23666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:36.635794 23666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:36.636058 23666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:36.643712 23666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b9d080d5114657aeee808e86a64312: Generated new TSK 0
I20260924 10:48:36.644297 23666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:36.656175 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23576
2026-09-24T10:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:36 dist-test-slave-6lwt krb5kdc[23675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:36 dist-test-slave-6lwt krb5kdc[23675](info): set up 2 sockets
Sep 24 10:48:36 dist-test-slave-6lwt krb5kdc[23675](info): commencing operation
krb5kdc: starting...
W20260924 10:48:37.733129 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:37 dist-test-slave-6lwt krb5kdc[23675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:37.851362 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:37.956236 23688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:37.956472 23688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:37.956511 23688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:37.960422 23688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:37.960471 23688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:37.960490 23688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:37.960506 23688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:37.960521 23688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:37.965548 23688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:37.966416 23688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:37.967299 23688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:37.975378 23693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:37.975494 23694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:37.975502 23696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:37.976047 23688 server_base.cc:1034] running on GCE node
I20260924 10:48:37.976637 23688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:37.977739 23688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:37.978931 23688 hybrid_clock.cc:648] HybridClock initialized: now 1790246917978905 us; error 42 us; skew 500 ppm
Sep 24 10:48:37 dist-test-slave-6lwt krb5kdc[23675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:37.981606 23688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:37.982470 23688 webserver.cc:460] Webserver started at http://127.0.0.1:45899/ using document root <none> and password file <none>
I20260924 10:48:37.983044 23688 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:37.983134 23688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:37.983336 23688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:37.985389 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "fc1aecb9676b4492b8ab89164ee615bb"
format_stamp: "Formatted at 2026-09-24 10:48:37 on dist-test-slave-6lwt"
server_key: "882aef7262d3204c6ae9b3724595f006"
server_key_iv: "110c82240e320ca17e911d518168f849"
server_key_version: "encryptionkey@0"
I20260924 10:48:37.985951 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "fc1aecb9676b4492b8ab89164ee615bb"
format_stamp: "Formatted at 2026-09-24 10:48:37 on dist-test-slave-6lwt"
server_key: "882aef7262d3204c6ae9b3724595f006"
server_key_iv: "110c82240e320ca17e911d518168f849"
server_key_version: "encryptionkey@0"
I20260924 10:48:37.989754 23688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:48:37.992005 23703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:37.993170 23688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:37.993301 23688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "fc1aecb9676b4492b8ab89164ee615bb"
format_stamp: "Formatted at 2026-09-24 10:48:37 on dist-test-slave-6lwt"
server_key: "882aef7262d3204c6ae9b3724595f006"
server_key_iv: "110c82240e320ca17e911d518168f849"
server_key_version: "encryptionkey@0"
I20260924 10:48:37.993405 23688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:38.007241 23688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:38.009865 23688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:38.010042 23688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:38.016629 23688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39585
I20260924 10:48:38.016655 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39585 every 8 connection(s)
I20260924 10:48:38.017549 23688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:38.020020 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:38.025192 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: Bootstrap starting.
I20260924 10:48:38.026559 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23688
I20260924 10:48:38.026697 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:38.026950 2776 external_mini_cluster.cc:1428] Setting key a200c55848f90a6640c399586fbfda2c
I20260924 10:48:38.027571 23756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:38.028412 23756 log.cc:824] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:38.030180 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: No bootstrap required, opened a new log
I20260924 10:48:38.033450 23756 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260924 10:48:38.033671 23756 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:38.033751 23756 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1aecb9676b4492b8ab89164ee615bb, State: Initialized, Role: FOLLOWER
I20260924 10:48:38.034155 23756 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260924 10:48:38.034299 23756 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:38.034384 23756 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:38.034498 23756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:48:38 dist-test-slave-6lwt krb5kdc[23675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:48:38.035564 23756 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260924 10:48:38.035880 23756 leader_election.cc:302] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1aecb9676b4492b8ab89164ee615bb; no voters:
I20260924 10:48:38.036206 23756 leader_election.cc:288] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:38.036274 23761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:38.036535 23761 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [term 1 LEADER]: Becoming Leader. State: Replica: fc1aecb9676b4492b8ab89164ee615bb, State: Running, Role: LEADER
I20260924 10:48:38.036860 23761 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260924 10:48:38.037425 23756 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:38.038453 23763 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1aecb9676b4492b8ab89164ee615bb. Latest consensus state: current_term: 1 leader_uuid: "fc1aecb9676b4492b8ab89164ee615bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } } }
I20260924 10:48:38.038586 23763 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:38.038893 23762 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1aecb9676b4492b8ab89164ee615bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1aecb9676b4492b8ab89164ee615bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } } }
I20260924 10:48:38.038991 23762 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:38.041937 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:38.027982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:59675 (local address 127.0.0.1:39585)
0924 10:48:38.028360 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:38.028374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:38.028405 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:48:38.029253 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:38.029272 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:38.029572 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:38.029892 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:38.030662 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:38.031603 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:38.032447 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:38.033024 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:38.033187 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:38.036186 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:38.036191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:38.036203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:38.036229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:38.038387 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:38.039043 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:38.039049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:38.039054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:38.039182 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:38.039615 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:38.039619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:38.039621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:38.040235 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:38.040458 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:38.040778 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 10:48:38.041033 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
W20260924 10:48:38.042165 23777 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:38.042258 23777 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:38.042366 23770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:38.043148 23770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:38.045097 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:38.048007 23770 catalog_manager.cc:1347] Generated new cluster ID: e15a0291209b430fbee6b551a93a9409
I20260924 10:48:38.048063 23770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:38.054441 23770 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:38.055552 23770 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:38.055724 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:38.047270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:38327 (local address 127.0.0.1:39585)
0924 10:48:38.047385 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:38.047390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:38.048307 (+ 917us) server_negotiation.cc:407] Connection header received
0924 10:48:38.048419 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:38.048422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:38.048461 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:38.048557 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:38.049190 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:38.049661 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:38.050428 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:38.050658 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:38.050760 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:38.052071 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:38.052075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:38.052078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:38.052100 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:38.053489 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:38.053949 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:38.053952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:38.053954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:38.054057 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:38.054454 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:38.054460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:38.054463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:38.054638 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:38.054771 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:38.055409 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 10:48:38.055497 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:48:38.064410 23770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1aecb9676b4492b8ab89164ee615bb: Generated new TSK 0
I20260924 10:48:38.065181 23770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:38.070490 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23688
W20260924 10:48:38.075531 23673 connection.cc:535] client connection to 127.0.0.1:39585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39585): Connection reset by peer (error 104)
2026-09-24T10:48:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:38 dist-test-slave-6lwt krb5kdc[23787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:38 dist-test-slave-6lwt krb5kdc[23787](info): set up 2 sockets
Sep 24 10:48:38 dist-test-slave-6lwt krb5kdc[23787](info): commencing operation
krb5kdc: starting...
W20260924 10:48:39.073531 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:39 dist-test-slave-6lwt krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:39.191952 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:39.303443 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:39.303726 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:39.303802 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:39.307811 23800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:39.307894 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:39.307931 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:39.307992 23800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:39.308032 23800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:39.313199 23800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:39.314224 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:39.315162 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:39.323196 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:39.323307 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:39.323196 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:39.323436 23800 server_base.cc:1034] running on GCE node
I20260924 10:48:39.323987 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:39.325090 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:39.326292 23800 hybrid_clock.cc:648] HybridClock initialized: now 1790246919326277 us; error 25 us; skew 500 ppm
Sep 24 10:48:39 dist-test-slave-6lwt krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:39.337641 23800 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:39.338572 23800 webserver.cc:460] Webserver started at http://127.0.0.1:36561/ using document root <none> and password file <none>
I20260924 10:48:39.339138 23800 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:39.339226 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:39.339462 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:39.341571 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "682174e6f15e4049b993a212b351dee8"
format_stamp: "Formatted at 2026-09-24 10:48:39 on dist-test-slave-6lwt"
server_key: "1548edbd28183f339fccf94b794c282b"
server_key_iv: "7be42b9feda9ba509a310a98868e11fb"
server_key_version: "encryptionkey@0"
I20260924 10:48:39.342212 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "682174e6f15e4049b993a212b351dee8"
format_stamp: "Formatted at 2026-09-24 10:48:39 on dist-test-slave-6lwt"
server_key: "1548edbd28183f339fccf94b794c282b"
server_key_iv: "7be42b9feda9ba509a310a98868e11fb"
server_key_version: "encryptionkey@0"
I20260924 10:48:39.346100 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:48:39.348502 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:39.349735 23800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:48:39.349833 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "682174e6f15e4049b993a212b351dee8"
format_stamp: "Formatted at 2026-09-24 10:48:39 on dist-test-slave-6lwt"
server_key: "1548edbd28183f339fccf94b794c282b"
server_key_iv: "7be42b9feda9ba509a310a98868e11fb"
server_key_version: "encryptionkey@0"
I20260924 10:48:39.349913 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:39.378316 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:39.381714 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:39.381875 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:39.389009 23800 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45347
I20260924 10:48:39.389081 23867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45347 every 8 connection(s)
I20260924 10:48:39.389950 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:39.392611 23868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:39.397953 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: Bootstrap starting.
I20260924 10:48:39.399258 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23800
I20260924 10:48:39.399359 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:39.399667 2776 external_mini_cluster.cc:1428] Setting key 3f62c79702321519b5e6d36153660201
I20260924 10:48:39.400588 23868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:39.401487 23868 log.cc:824] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:39.403514 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: No bootstrap required, opened a new log
I20260924 10:48:39.407246 23868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260924 10:48:39.407481 23868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:39.407583 23868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682174e6f15e4049b993a212b351dee8, State: Initialized, Role: FOLLOWER
I20260924 10:48:39.408058 23868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260924 10:48:39.408211 23868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:39.408288 23868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:39.408390 23868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:39.409420 23868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260924 10:48:39.409744 23868 leader_election.cc:302] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 682174e6f15e4049b993a212b351dee8; no voters:
I20260924 10:48:39.409929 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:39.400735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:58361 (local address 127.0.0.1:45347)
0924 10:48:39.401225 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:39.401243 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:39.401289 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:48:39.402070 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:39.402087 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:39.402507 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:39.402844 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:39.403243 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:39.404697 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:39.405599 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:39.406916 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:39.407077 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:39.407976 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:39.407981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:39.407995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:39.408488 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:39.408558 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:39.408589 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:39.408716 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:39.408846 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 10:48:39.408994 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":158,"threads_started":1}
I20260924 10:48:39.410044 23868 leader_election.cc:288] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:39.410145 23873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:39.410349 23873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [term 1 LEADER]: Becoming Leader. State: Replica: 682174e6f15e4049b993a212b351dee8, State: Running, Role: LEADER
I20260924 10:48:39.410712 23873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260924 10:48:39.411171 23868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:39.412763 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:39.412750 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "682174e6f15e4049b993a212b351dee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } } }
I20260924 10:48:39.412854 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:39.413166 23875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682174e6f15e4049b993a212b351dee8. Latest consensus state: current_term: 1 leader_uuid: "682174e6f15e4049b993a212b351dee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682174e6f15e4049b993a212b351dee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } } }
I20260924 10:48:39.415028 23875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8 [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:39.416028 23888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:39.416123 23888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:39.416200 23887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:39.417160 23887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:39.420557 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:39.414965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:38181 (local address 127.0.0.1:45347)
0924 10:48:39.415174 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:39.415178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:39.416517 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 10:48:39.416632 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:39.416635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:39.416678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:39.416759 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:39.417093 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:39.417883 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:39.418970 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:39.419195 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:39.419306 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:39.420120 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:39.420124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:39.420127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:39.420144 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:39.420166 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:39.420175 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:39.420268 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:39.420385 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 10:48:39.420417 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260924 10:48:39.423097 23887 catalog_manager.cc:1347] Generated new cluster ID: e4a4f5a0bb5648358ffff35d2a5957d7
I20260924 10:48:39.423275 23887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:39.434664 23887 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:39.435498 23887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:39.445644 23887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 682174e6f15e4049b993a212b351dee8: Generated new TSK 0
I20260924 10:48:39.446300 23887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:39.450313 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23800
W20260924 10:48:39.457625 23785 connection.cc:535] client connection to 127.0.0.1:45347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45347): Connection reset by peer (error 104)
2026-09-24T10:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:39 dist-test-slave-6lwt krb5kdc[23899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:39 dist-test-slave-6lwt krb5kdc[23899](info): set up 2 sockets
Sep 24 10:48:39 dist-test-slave-6lwt krb5kdc[23899](info): commencing operation
krb5kdc: starting...
W20260924 10:48:40.501441 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:40 dist-test-slave-6lwt krb5kdc[23899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:40.620596 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:40.725234 23912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:40.725482 23912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:40.725521 23912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:40.729475 23912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:40.729526 23912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:40.729545 23912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:40.729561 23912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:40.729594 23912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:40.734632 23912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:40.735580 23912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:40.736558 23912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:40.744787 23917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:40.744985 23912 server_base.cc:1034] running on GCE node
W20260924 10:48:40.744761 23918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:40.744761 23920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:40.745703 23912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:40.746752 23912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:40.747970 23912 hybrid_clock.cc:648] HybridClock initialized: now 1790246920747959 us; error 23 us; skew 500 ppm
Sep 24 10:48:40 dist-test-slave-6lwt krb5kdc[23899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:40.750567 23912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:40.751398 23912 webserver.cc:460] Webserver started at http://127.0.0.1:35369/ using document root <none> and password file <none>
I20260924 10:48:40.751935 23912 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:40.752024 23912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:40.752249 23912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:40.754457 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "9d239eb8180a42a5a848ee6dd969350c"
format_stamp: "Formatted at 2026-09-24 10:48:40 on dist-test-slave-6lwt"
server_key: "e2f28fc506321b6a29c5a7e893be25aa"
server_key_iv: "adfbfc6f6b8ed25b90b528d44633a779"
server_key_version: "encryptionkey@0"
I20260924 10:48:40.755061 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "9d239eb8180a42a5a848ee6dd969350c"
format_stamp: "Formatted at 2026-09-24 10:48:40 on dist-test-slave-6lwt"
server_key: "e2f28fc506321b6a29c5a7e893be25aa"
server_key_iv: "adfbfc6f6b8ed25b90b528d44633a779"
server_key_version: "encryptionkey@0"
I20260924 10:48:40.759110 23912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:48:40.761333 23927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:40.762467 23912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:40.762595 23912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "9d239eb8180a42a5a848ee6dd969350c"
format_stamp: "Formatted at 2026-09-24 10:48:40 on dist-test-slave-6lwt"
server_key: "e2f28fc506321b6a29c5a7e893be25aa"
server_key_iv: "adfbfc6f6b8ed25b90b528d44633a779"
server_key_version: "encryptionkey@0"
I20260924 10:48:40.762697 23912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:40.767110 23912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:40.769971 23912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:40.770152 23912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:40.776809 23912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43035
I20260924 10:48:40.776818 23979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43035 every 8 connection(s)
I20260924 10:48:40.777698 23912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:40.780158 23980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:40.785233 23980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c: Bootstrap starting.
I20260924 10:48:40.785682 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 23912
I20260924 10:48:40.785804 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:40.786075 2776 external_mini_cluster.cc:1428] Setting key c8d8a5ef2c18314003ef8dc2b9940f80
I20260924 10:48:40.787762 23980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:40.788427 23980 log.cc:824] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:40.790369 23980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c: No bootstrap required, opened a new log
I20260924 10:48:40.790881 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:40.787251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:53133 (local address 127.0.0.1:43035)
0924 10:48:40.787663 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:40.787677 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:40.787709 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:48:40.788311 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:40.788326 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:40.788641 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:40.789003 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:40.789722 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:40.789727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:40.789748 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:40.789758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:40.789803 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:40.789821 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:40.789926 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:40.790023 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:48:40.790165 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":155,"threads_started":1}
I20260924 10:48:40.793619 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:40.794342 23980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } }
I20260924 10:48:40.794528 23980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:40.794590 23980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d239eb8180a42a5a848ee6dd969350c, State: Initialized, Role: FOLLOWER
I20260924 10:48:40.794994 23980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } }
I20260924 10:48:40.795106 23980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:40.795184 23980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:40.795295 23980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:40.796438 23980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } }
I20260924 10:48:40.796835 23980 leader_election.cc:302] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d239eb8180a42a5a848ee6dd969350c; no voters:
I20260924 10:48:40.797156 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:40.795509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:43145 (local address 127.0.0.1:43035)
0924 10:48:40.795637 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:40.795641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:40.795922 (+ 281us) server_negotiation.cc:407] Connection header received
0924 10:48:40.795996 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:40.795998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:40.796023 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:40.796083 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:40.796730 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:40.796733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:40.796735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:40.796744 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:40.796765 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:40.796775 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:40.796841 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:40.796963 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 10:48:40.797003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:48:40.797185 23980 leader_election.cc:288] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:40.797256 23990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:40.797477 23990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [term 1 LEADER]: Becoming Leader. State: Replica: 9d239eb8180a42a5a848ee6dd969350c, State: Running, Role: LEADER
I20260924 10:48:40.797787 23990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } }
I20260924 10:48:40.798384 23980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:40.799284 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d239eb8180a42a5a848ee6dd969350c. Latest consensus state: current_term: 1 leader_uuid: "9d239eb8180a42a5a848ee6dd969350c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } } }
I20260924 10:48:40.799458 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:40.799800 23999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:40.799764 23991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d239eb8180a42a5a848ee6dd969350c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d239eb8180a42a5a848ee6dd969350c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43035 } } }
I20260924 10:48:40.799862 23991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:40.802428 23999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:40.807960 23999 catalog_manager.cc:1347] Generated new cluster ID: 7f9c2b55cbe54d6794d7808ab192879b
I20260924 10:48:40.808037 23999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:40.823300 23999 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:40.823589 23999 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:40.831295 23999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d239eb8180a42a5a848ee6dd969350c: Generated new TSK 0
I20260924 10:48:40.831924 23999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:40.840080 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 23912
2026-09-24T10:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:40 dist-test-slave-6lwt krb5kdc[24011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:40 dist-test-slave-6lwt krb5kdc[24011](info): set up 2 sockets
Sep 24 10:48:40 dist-test-slave-6lwt krb5kdc[24011](info): commencing operation
krb5kdc: starting...
W20260924 10:48:41.845741 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:41 dist-test-slave-6lwt krb5kdc[24011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:41.963099 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:42.069455 24024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:42.069696 24024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:42.069737 24024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:42.073697 24024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:42.073747 24024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:42.073765 24024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:42.073807 24024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:42.073851 24024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:42.079083 24024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:42.080070 24024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:42.081118 24024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:42.089092 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:42.089105 24029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:42.089102 24030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:42.089174 24024 server_base.cc:1034] running on GCE node
I20260924 10:48:42.089795 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:42.090793 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:42.091955 24024 hybrid_clock.cc:648] HybridClock initialized: now 1790246922091931 us; error 40 us; skew 500 ppm
Sep 24 10:48:42 dist-test-slave-6lwt krb5kdc[24011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:42.094693 24024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:42.095549 24024 webserver.cc:460] Webserver started at http://127.0.0.1:40635/ using document root <none> and password file <none>
I20260924 10:48:42.096077 24024 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:42.096167 24024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:42.096392 24024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:42.098443 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "d60ad664594947ecbdf9e74dcab98bc6"
format_stamp: "Formatted at 2026-09-24 10:48:42 on dist-test-slave-6lwt"
server_key: "9d315e597b3907f3d13cb72ab4b9e1b3"
server_key_iv: "89c7128518cd674a43fba3150b5d2baa"
server_key_version: "encryptionkey@0"
I20260924 10:48:42.098994 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "d60ad664594947ecbdf9e74dcab98bc6"
format_stamp: "Formatted at 2026-09-24 10:48:42 on dist-test-slave-6lwt"
server_key: "9d315e597b3907f3d13cb72ab4b9e1b3"
server_key_iv: "89c7128518cd674a43fba3150b5d2baa"
server_key_version: "encryptionkey@0"
I20260924 10:48:42.102511 24024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:48:42.104737 24039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:42.105906 24024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:48:42.106038 24024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "d60ad664594947ecbdf9e74dcab98bc6"
format_stamp: "Formatted at 2026-09-24 10:48:42 on dist-test-slave-6lwt"
server_key: "9d315e597b3907f3d13cb72ab4b9e1b3"
server_key_iv: "89c7128518cd674a43fba3150b5d2baa"
server_key_version: "encryptionkey@0"
I20260924 10:48:42.106143 24024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:42.118902 24024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:42.121439 24024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:42.121619 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:42.128209 24024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41085
I20260924 10:48:42.128245 24091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41085 every 8 connection(s)
I20260924 10:48:42.129024 24024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:42.131410 24092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:42.136270 24092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6: Bootstrap starting.
I20260924 10:48:42.138451 24092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:42.138617 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24024
I20260924 10:48:42.138764 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:42.139053 2776 external_mini_cluster.cc:1428] Setting key b71b747351132dd9fb169d009e93cb99
I20260924 10:48:42.139250 24092 log.cc:824] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:42.141121 24092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6: No bootstrap required, opened a new log
I20260924 10:48:42.144304 24092 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260924 10:48:42.144482 24092 raft_consensus.cc:374] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:42.144538 24092 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60ad664594947ecbdf9e74dcab98bc6, State: Initialized, Role: FOLLOWER
I20260924 10:48:42.144965 24092 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260924 10:48:42.145098 24092 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:42.145165 24092 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:42.145258 24092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:42.146114 24092 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260924 10:48:42.146461 24092 leader_election.cc:302] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60ad664594947ecbdf9e74dcab98bc6; no voters:
I20260924 10:48:42.146764 24092 leader_election.cc:288] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:42.146842 24097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 10:48:42 dist-test-slave-6lwt krb5kdc[24011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:48:42.147069 24097 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [term 1 LEADER]: Becoming Leader. State: Replica: d60ad664594947ecbdf9e74dcab98bc6, State: Running, Role: LEADER
I20260924 10:48:42.147396 24097 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260924 10:48:42.147922 24092 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:42.149248 24098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d60ad664594947ecbdf9e74dcab98bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260924 10:48:42.149376 24098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:42.149806 24106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:42.150159 24099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60ad664594947ecbdf9e74dcab98bc6. Latest consensus state: current_term: 1 leader_uuid: "d60ad664594947ecbdf9e74dcab98bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60ad664594947ecbdf9e74dcab98bc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260924 10:48:42.150328 24099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:42.153258 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:42.140046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:53939 (local address 127.0.0.1:41085)
0924 10:48:42.140415 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:42.140428 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:42.140459 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:48:42.141109 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:42.141125 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:42.141473 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:42.141788 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:42.142568 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:42.143469 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:42.144312 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:42.144882 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:42.145050 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:42.147723 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:42.147728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:42.147742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:42.147768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:42.149909 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:42.150578 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:42.150583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:42.150588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:42.150722 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:42.151098 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:42.151101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:42.151103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:42.151563 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:42.151795 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:42.152134 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:48:42.152357 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260924 10:48:42.153373 24106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:42.156057 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:42.158880 24106 catalog_manager.cc:1347] Generated new cluster ID: 90fb1c6c27434bdf92320dc3610de993
I20260924 10:48:42.158937 24106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:42.166757 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:42.158008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:37995 (local address 127.0.0.1:41085)
0924 10:48:42.158100 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:42.158104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:42.158278 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:48:42.158394 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:42.158397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:42.158435 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:42.158521 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:42.159231 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:42.160437 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:42.161508 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:42.161861 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:42.161969 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:42.163434 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:42.163438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:42.163441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:42.163465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:42.165080 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:42.165549 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:42.165552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:42.165554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:42.165632 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:42.165935 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:42.165938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:42.165939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:42.166148 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:42.166269 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:42.166537 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:48:42.166626 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 10:48:42.171593 24106 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:42.172382 24106 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:42.178953 24106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d60ad664594947ecbdf9e74dcab98bc6: Generated new TSK 0
I20260924 10:48:42.179531 24106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:42.186050 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24024
W20260924 10:48:42.190986 24009 connection.cc:535] client connection to 127.0.0.1:41085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41085): Connection reset by peer (error 104)
2026-09-24T10:48:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:42 dist-test-slave-6lwt krb5kdc[24123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:42 dist-test-slave-6lwt krb5kdc[24123](info): set up 2 sockets
Sep 24 10:48:42 dist-test-slave-6lwt krb5kdc[24123](info): commencing operation
krb5kdc: starting...
W20260924 10:48:43.189415 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:43 dist-test-slave-6lwt krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:43.307276 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:43.409857 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:43.410127 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:43.410194 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:43.414098 24136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:43.414168 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:43.414206 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:43.414256 24136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:43.414290 24136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:43.419104 24136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:43.420305 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:43.421207 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:43.428728 24144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:43.428830 24142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:43.428923 24136 server_base.cc:1034] running on GCE node
W20260924 10:48:43.428741 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:43.429559 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:43.430444 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:43.431593 24136 hybrid_clock.cc:648] HybridClock initialized: now 1790246923431574 us; error 30 us; skew 500 ppm
Sep 24 10:48:43 dist-test-slave-6lwt krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:43.434128 24136 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:43.434922 24136 webserver.cc:460] Webserver started at http://127.0.0.1:41999/ using document root <none> and password file <none>
I20260924 10:48:43.435422 24136 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:43.435478 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:43.435695 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:43.437578 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "46d65499839e43928015e36b6e8a00d1"
format_stamp: "Formatted at 2026-09-24 10:48:43 on dist-test-slave-6lwt"
server_key: "e2d1d0fa15d59cec826c3c1374d89577"
server_key_iv: "ba966ecbeb9ffd1b5566726537dbc40a"
server_key_version: "encryptionkey@0"
I20260924 10:48:43.438104 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "46d65499839e43928015e36b6e8a00d1"
format_stamp: "Formatted at 2026-09-24 10:48:43 on dist-test-slave-6lwt"
server_key: "e2d1d0fa15d59cec826c3c1374d89577"
server_key_iv: "ba966ecbeb9ffd1b5566726537dbc40a"
server_key_version: "encryptionkey@0"
I20260924 10:48:43.441350 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:43.443496 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:43.444609 24136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:43.444731 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "46d65499839e43928015e36b6e8a00d1"
format_stamp: "Formatted at 2026-09-24 10:48:43 on dist-test-slave-6lwt"
server_key: "e2d1d0fa15d59cec826c3c1374d89577"
server_key_iv: "ba966ecbeb9ffd1b5566726537dbc40a"
server_key_version: "encryptionkey@0"
I20260924 10:48:43.444839 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:43.457801 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:43.460065 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:43.460232 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:43.467046 24136 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40653
I20260924 10:48:43.467072 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40653 every 8 connection(s)
I20260924 10:48:43.467881 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:43.470342 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:43.472877 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24136
I20260924 10:48:43.473053 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:43.473356 2776 external_mini_cluster.cc:1428] Setting key c8fbfad03fffb6c6a84616395ef2bf5d
I20260924 10:48:43.475992 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: Bootstrap starting.
I20260924 10:48:43.478009 24204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:43.478704 24204 log.cc:824] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:43.480655 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: No bootstrap required, opened a new log
I20260924 10:48:43.482132 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:43.474629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:42777 (local address 127.0.0.1:40653)
0924 10:48:43.475024 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:43.475038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:43.475068 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:48:43.475616 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:43.475631 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:43.475919 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:43.476226 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:43.476866 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:43.477841 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:43.478748 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:43.479223 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:43.479376 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:43.480307 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:43.480313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:43.480328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:43.480795 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:43.480859 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:43.480905 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:43.481050 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:43.481175 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:48:43.481296 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260924 10:48:43.484885 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:43.484840 24204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260924 10:48:43.485155 24204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:43.485241 24204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d65499839e43928015e36b6e8a00d1, State: Initialized, Role: FOLLOWER
I20260924 10:48:43.485677 24204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260924 10:48:43.485831 24204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:43.485894 24204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:43.485994 24204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:43.486874 24204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260924 10:48:43.487289 24204 leader_election.cc:302] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d65499839e43928015e36b6e8a00d1; no voters:
I20260924 10:48:43.487632 24204 leader_election.cc:288] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:43.487699 24214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:43.487957 24214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [term 1 LEADER]: Becoming Leader. State: Replica: 46d65499839e43928015e36b6e8a00d1, State: Running, Role: LEADER
I20260924 10:48:43.488308 24214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260924 10:48:43.488852 24204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:43.489540 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d65499839e43928015e36b6e8a00d1. Latest consensus state: current_term: 1 leader_uuid: "46d65499839e43928015e36b6e8a00d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } } }
I20260924 10:48:43.489650 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:43.490295 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d65499839e43928015e36b6e8a00d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d65499839e43928015e36b6e8a00d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } } }
I20260924 10:48:43.490376 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:43.491811 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:43.487038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:57519 (local address 127.0.0.1:40653)
0924 10:48:43.487144 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:43.487148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:43.487331 (+ 183us) server_negotiation.cc:407] Connection header received
0924 10:48:43.487482 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:43.487485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:43.487533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:43.487621 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:43.487934 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:43.488693 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:43.489926 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:43.490270 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:43.490382 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:43.491436 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:43.491438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:43.491440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:43.491450 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:43.491470 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:43.491479 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:43.491547 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:43.491695 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 10:48:43.491723 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 10:48:43.493997 24229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:43.494097 24229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:43.494187 24222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:43.494848 24222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:43.500442 24222 catalog_manager.cc:1347] Generated new cluster ID: 0c2e3f61de9e49d1947f659c4c14f15b
I20260924 10:48:43.500504 24222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:43.515751 24222 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:43.517006 24222 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:43.526849 24222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d65499839e43928015e36b6e8a00d1: Generated new TSK 0
I20260924 10:48:43.527566 24222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:43.539732 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24136
W20260924 10:48:43.545295 24121 connection.cc:535] client connection to 127.0.0.1:40653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40653): Connection reset by peer (error 104)
2026-09-24T10:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:43 dist-test-slave-6lwt krb5kdc[24235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:43 dist-test-slave-6lwt krb5kdc[24235](info): set up 2 sockets
Sep 24 10:48:43 dist-test-slave-6lwt krb5kdc[24235](info): commencing operation
krb5kdc: starting...
W20260924 10:48:44.561350 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:44 dist-test-slave-6lwt krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:48:44.675824 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:44.781947 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:44.782181 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:44.782222 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:44.786121 24248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:44.786168 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:44.786187 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:44.786202 24248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:44.786216 24248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:44.791193 24248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:44.792068 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:44.793022 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:44.800875 24256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:44.800930 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:44.800930 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:44.801041 24248 server_base.cc:1034] running on GCE node
I20260924 10:48:44.801622 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:44.802659 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:44.803843 24248 hybrid_clock.cc:648] HybridClock initialized: now 1790246924803834 us; error 26 us; skew 500 ppm
Sep 24 10:48:44 dist-test-slave-6lwt krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:44.813160 24248 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:48:44.813982 24248 webserver.cc:460] Webserver started at http://127.0.0.1:33507/ using document root <none> and password file <none>
I20260924 10:48:44.814513 24248 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:44.814597 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:44.814811 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:44.816790 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "648a2a72dba2422a92bd3f9e717f3642"
format_stamp: "Formatted at 2026-09-24 10:48:44 on dist-test-slave-6lwt"
server_key: "ee1d81057dbafe2f841939327ba003d4"
server_key_iv: "c92f147208d7da5c70aac6ff7fe5ddd9"
server_key_version: "encryptionkey@0"
I20260924 10:48:44.817400 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "648a2a72dba2422a92bd3f9e717f3642"
format_stamp: "Formatted at 2026-09-24 10:48:44 on dist-test-slave-6lwt"
server_key: "ee1d81057dbafe2f841939327ba003d4"
server_key_iv: "c92f147208d7da5c70aac6ff7fe5ddd9"
server_key_version: "encryptionkey@0"
I20260924 10:48:44.821013 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:44.823251 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:44.824376 24248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:44.824512 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "648a2a72dba2422a92bd3f9e717f3642"
format_stamp: "Formatted at 2026-09-24 10:48:44 on dist-test-slave-6lwt"
server_key: "ee1d81057dbafe2f841939327ba003d4"
server_key_iv: "c92f147208d7da5c70aac6ff7fe5ddd9"
server_key_version: "encryptionkey@0"
I20260924 10:48:44.824608 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:44.828845 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:44.831465 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:44.831642 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:44.838064 24248 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39139
I20260924 10:48:44.838089 24315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39139 every 8 connection(s)
I20260924 10:48:44.838891 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:44.840657 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24248
I20260924 10:48:44.840788 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:44.841087 2776 external_mini_cluster.cc:1428] Setting key c437ab2f5790d405ae331318518a29fe
I20260924 10:48:44.841573 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:44.846372 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:44.842403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:35845 (local address 127.0.0.1:39139)
0924 10:48:44.842800 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:44.842830 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:44.842873 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:48:44.843623 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:44.843644 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:44.843991 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:44.844341 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:44.845049 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:44.845054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:44.845068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:44.845081 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:44.845140 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:44.845164 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:44.845306 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:44.845373 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 10:48:44.845511 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260924 10:48:44.848373 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642: Bootstrap starting.
I20260924 10:48:44.849026 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:44.850489 24316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:44.851277 24316 log.cc:824] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:44.852583 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:44.850937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:36299 (local address 127.0.0.1:39139)
0924 10:48:44.851055 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:44.851059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:44.851265 (+ 206us) server_negotiation.cc:407] Connection header received
0924 10:48:44.851388 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:44.851391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:44.851425 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:44.851499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:44.852225 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:44.852228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:44.852230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:44.852239 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:44.852260 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:44.852269 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:44.852335 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:44.852432 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:48:44.852460 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:48:44.853468 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642: No bootstrap required, opened a new log
I20260924 10:48:44.856544 24316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } }
I20260924 10:48:44.856775 24316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:44.856830 24316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648a2a72dba2422a92bd3f9e717f3642, State: Initialized, Role: FOLLOWER
I20260924 10:48:44.857210 24316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } }
I20260924 10:48:44.857373 24316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:44.857437 24316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:44.857532 24316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:44.858515 24316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } }
I20260924 10:48:44.858842 24316 leader_election.cc:302] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 648a2a72dba2422a92bd3f9e717f3642; no voters:
I20260924 10:48:44.859094 24316 leader_election.cc:288] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:44.859200 24326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:44.859406 24326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [term 1 LEADER]: Becoming Leader. State: Replica: 648a2a72dba2422a92bd3f9e717f3642, State: Running, Role: LEADER
I20260924 10:48:44.859720 24326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } }
I20260924 10:48:44.860196 24316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:44.861296 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648a2a72dba2422a92bd3f9e717f3642. Latest consensus state: current_term: 1 leader_uuid: "648a2a72dba2422a92bd3f9e717f3642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } } }
I20260924 10:48:44.861433 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:44.861819 24335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:44.864317 24335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:44.861788 24327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "648a2a72dba2422a92bd3f9e717f3642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648a2a72dba2422a92bd3f9e717f3642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39139 } } }
I20260924 10:48:44.864951 24327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:44.869112 24335 catalog_manager.cc:1347] Generated new cluster ID: d90a12672f4147e3b549bc78c22d3685
I20260924 10:48:44.869208 24335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:44.876516 24335 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:44.876754 24335 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:44.882483 24335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 648a2a72dba2422a92bd3f9e717f3642: Generated new TSK 0
I20260924 10:48:44.883136 24335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:44.887986 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24248
2026-09-24T10:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:44 dist-test-slave-6lwt krb5kdc[24347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:44 dist-test-slave-6lwt krb5kdc[24347](info): set up 2 sockets
Sep 24 10:48:44 dist-test-slave-6lwt krb5kdc[24347](info): commencing operation
krb5kdc: starting...
W20260924 10:48:45.921342 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:46.039965 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42199
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:42199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:46.150913 24360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:46.151171 24360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:46.151244 24360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:46.155198 24360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:46.155280 24360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:46.155318 24360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:46.155373 24360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:46.155411 24360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:46.160360 24360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:42199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:46.161388 24360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:46.162387 24360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:46.170619 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:46.170648 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:46.170728 24360 server_base.cc:1034] running on GCE node
W20260924 10:48:46.170666 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:46.171481 24360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:46.172514 24360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:46.173694 24360 hybrid_clock.cc:648] HybridClock initialized: now 1790246926173680 us; error 26 us; skew 500 ppm
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246926, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:46.176288 24360 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:46.177184 24360 webserver.cc:460] Webserver started at http://127.2.182.62:43409/ using document root <none> and password file <none>
I20260924 10:48:46.177680 24360 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:46.177736 24360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:46.177906 24360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:46.179872 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "7435d8ca8a0e4981bb76915e8ad050f8"
format_stamp: "Formatted at 2026-09-24 10:48:46 on dist-test-slave-6lwt"
server_key: "5250754c82596fa66f60c9492eb5d1a3"
server_key_iv: "9acbf01cde198672d166b4bf341f7a4e"
server_key_version: "encryptionkey@0"
I20260924 10:48:46.180366 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "7435d8ca8a0e4981bb76915e8ad050f8"
format_stamp: "Formatted at 2026-09-24 10:48:46 on dist-test-slave-6lwt"
server_key: "5250754c82596fa66f60c9492eb5d1a3"
server_key_iv: "9acbf01cde198672d166b4bf341f7a4e"
server_key_version: "encryptionkey@0"
I20260924 10:48:46.183919 24360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:46.186128 24375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:46.187282 24360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:46.187398 24360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "7435d8ca8a0e4981bb76915e8ad050f8"
format_stamp: "Formatted at 2026-09-24 10:48:46 on dist-test-slave-6lwt"
server_key: "5250754c82596fa66f60c9492eb5d1a3"
server_key_iv: "9acbf01cde198672d166b4bf341f7a4e"
server_key_version: "encryptionkey@0"
I20260924 10:48:46.187497 24360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:46.199954 24360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:46.203058 24360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:46.203198 24360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:46.209771 24360 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:42199
I20260924 10:48:46.209797 24427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:42199 every 8 connection(s)
I20260924 10:48:46.210671 24360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:46.213220 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:46.215278 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24360
I20260924 10:48:46.215404 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:46.215713 2776 external_mini_cluster.cc:1428] Setting key 787a5f66a873458c454ae363049ffb89
I20260924 10:48:46.218972 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8: Bootstrap starting.
I20260924 10:48:46.220927 24428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:46.221622 24428 log.cc:824] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:46.223407 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8: No bootstrap required, opened a new log
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:46.226747 24428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } }
I20260924 10:48:46.226938 24428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:46.227001 24428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7435d8ca8a0e4981bb76915e8ad050f8, State: Initialized, Role: FOLLOWER
I20260924 10:48:46.227468 24428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } }
I20260924 10:48:46.227612 24428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:46.227692 24428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:46.227803 24428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:46.228713 24428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } }
I20260924 10:48:46.229072 24428 leader_election.cc:302] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7435d8ca8a0e4981bb76915e8ad050f8; no voters:
I20260924 10:48:46.229362 24428 leader_election.cc:288] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:46.229477 24433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:46.229693 24433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7435d8ca8a0e4981bb76915e8ad050f8, State: Running, Role: LEADER
I20260924 10:48:46.230018 24433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } }
I20260924 10:48:46.230293 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:46.217033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55035 (local address 127.2.182.62:42199)
0924 10:48:46.217469 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:46.217483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:46.217517 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:48:46.218204 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:46.218220 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:46.218545 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:46.218873 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:46.219628 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:46.220526 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:46.221472 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:46.221994 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:46.222170 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:46.224794 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:46.224799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:46.224809 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:46.224840 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:46.227031 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:46.227567 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:46.227573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:46.227578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:46.227719 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:46.228136 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:46.228139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:46.228140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:46.228484 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:46.228663 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:46.229073 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 10:48:46.229309 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260924 10:48:46.230536 24428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:46.231117 24433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7435d8ca8a0e4981bb76915e8ad050f8. Latest consensus state: current_term: 1 leader_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } } }
I20260924 10:48:46.231281 24433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:46.231540 24435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7435d8ca8a0e4981bb76915e8ad050f8" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42199 } } }
I20260924 10:48:46.231686 24435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:46.231673 24441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:46.233311 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:46.234822 24441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:46.240185 24441 catalog_manager.cc:1347] Generated new cluster ID: 004b58ffd538455fbf70f65b008b012b
I20260924 10:48:46.240288 24441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:46.244558 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:46.235405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35741 (local address 127.2.182.62:42199)
0924 10:48:46.235498 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:46.235502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:46.235814 (+ 312us) server_negotiation.cc:407] Connection header received
0924 10:48:46.235924 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:46.235927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:46.235969 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:46.236045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:46.236811 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:46.237548 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:46.238739 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:46.239151 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:46.239245 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:46.241040 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:46.241044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:46.241046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:46.241066 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:46.242607 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:46.243154 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:46.243157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:46.243159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:46.243233 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:46.243592 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:46.243595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:46.243597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:46.243771 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:46.243908 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:46.244179 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:48:46.244264 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":37}
I20260924 10:48:46.249552 24441 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:46.250645 24441 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:46.267704 24441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7435d8ca8a0e4981bb76915e8ad050f8: Generated new TSK 0
I20260924 10:48:46.268426 24441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:46.274099 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24360
W20260924 10:48:46.279548 24345 connection.cc:535] client connection to 127.2.182.62:42199 recv error: Network error: failed to read from TLS socket (remote: 127.2.182.62:42199): Connection reset by peer (error 104)
2026-09-24T10:48:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24459](info): set up 2 sockets
Sep 24 10:48:46 dist-test-slave-6lwt krb5kdc[24459](info): commencing operation
krb5kdc: starting...
W20260924 10:48:47.289361 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:47 dist-test-slave-6lwt krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:47.405885 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42089
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:47.510485 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:47.510704 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:47.510742 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:47.514629 24472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:47.514679 24472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:47.514698 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:47.514714 24472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:47.514729 24472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:47.519735 24472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:42089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:47.520773 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:47.521713 24472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:47.529714 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:47.529723 24477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:47.529846 24472 server_base.cc:1034] running on GCE node
W20260924 10:48:47.529726 24478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:47.530484 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:47.531469 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:47.532676 24472 hybrid_clock.cc:648] HybridClock initialized: now 1790246927532642 us; error 44 us; skew 500 ppm
Sep 24 10:48:47 dist-test-slave-6lwt krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:47.535348 24472 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:47.536183 24472 webserver.cc:460] Webserver started at http://127.2.182.62:36639/ using document root <none> and password file <none>
I20260924 10:48:47.536671 24472 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:47.536727 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:47.536937 24472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:47.538857 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "92d248cba0784fb4aa3548f2f4858a91"
format_stamp: "Formatted at 2026-09-24 10:48:47 on dist-test-slave-6lwt"
server_key: "135a4376f01ac89ca2eaf68a9f02971d"
server_key_iv: "dc00847fd88889b14cc384aa2c2685c5"
server_key_version: "encryptionkey@0"
I20260924 10:48:47.539347 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "92d248cba0784fb4aa3548f2f4858a91"
format_stamp: "Formatted at 2026-09-24 10:48:47 on dist-test-slave-6lwt"
server_key: "135a4376f01ac89ca2eaf68a9f02971d"
server_key_iv: "dc00847fd88889b14cc384aa2c2685c5"
server_key_version: "encryptionkey@0"
I20260924 10:48:47.542919 24472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:47.545143 24487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:47.546347 24472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:47.546446 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "92d248cba0784fb4aa3548f2f4858a91"
format_stamp: "Formatted at 2026-09-24 10:48:47 on dist-test-slave-6lwt"
server_key: "135a4376f01ac89ca2eaf68a9f02971d"
server_key_iv: "dc00847fd88889b14cc384aa2c2685c5"
server_key_version: "encryptionkey@0"
I20260924 10:48:47.546540 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:47.560415 24472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:47.563090 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:47.563236 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:47.570227 24472 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:42089
I20260924 10:48:47.570253 24539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:42089 every 8 connection(s)
I20260924 10:48:47.571156 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:47.573727 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:47.578860 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: Bootstrap starting.
I20260924 10:48:47.581171 24540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:47.581208 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24472
I20260924 10:48:47.581359 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:47.581624 2776 external_mini_cluster.cc:1428] Setting key 3970695cda30e2b688c0dca0b528bd37
I20260924 10:48:47.581944 24540 log.cc:824] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:47.583812 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: No bootstrap required, opened a new log
I20260924 10:48:47.587728 24540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } }
I20260924 10:48:47.587972 24540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:47.588047 24540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d248cba0784fb4aa3548f2f4858a91, State: Initialized, Role: FOLLOWER
I20260924 10:48:47.588524 24540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } }
I20260924 10:48:47.588660 24540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:47.588745 24540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:47.588840 24540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:47.589833 24540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } }
I20260924 10:48:47.590154 24540 leader_election.cc:302] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92d248cba0784fb4aa3548f2f4858a91; no voters:
I20260924 10:48:47.590384 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:47.582709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33653 (local address 127.2.182.62:42089)
0924 10:48:47.583054 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:47.583078 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:47.583110 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:48:47.583701 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:47.583716 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:47.584084 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:47.584345 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:47.584681 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:47.585684 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:47.586970 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:47.587406 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:47.587515 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:47.588474 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:47.588479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:47.588491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:47.589040 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:47.589091 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:47.589112 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:47.589211 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:47.589433 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 10:48:47.589584 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
I20260924 10:48:47.590461 24540 leader_election.cc:288] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:47.590556 24545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:47.591624 24540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:47.591709 24545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [term 1 LEADER]: Becoming Leader. State: Replica: 92d248cba0784fb4aa3548f2f4858a91, State: Running, Role: LEADER
I20260924 10:48:47.591987 24545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } }
I20260924 10:48:47.593624 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:47.593926 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92d248cba0784fb4aa3548f2f4858a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } } }
I20260924 10:48:47.594043 24545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d248cba0784fb4aa3548f2f4858a91. Latest consensus state: current_term: 1 leader_uuid: "92d248cba0784fb4aa3548f2f4858a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d248cba0784fb4aa3548f2f4858a91" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 42089 } } }
I20260924 10:48:47.596067 24545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:47.596063 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91 [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:47.596765 24559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:47.596879 24559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:47.597019 24567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:47.597667 24567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:47.601531 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:47.595741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36995 (local address 127.2.182.62:42089)
0924 10:48:47.595840 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:47.595844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:47.596540 (+ 696us) server_negotiation.cc:407] Connection header received
0924 10:48:47.596631 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:47.596634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:47.596676 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:47.596752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:47.597169 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:47.597930 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:47.599098 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:47.599445 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:47.599519 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:47.600577 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:47.600580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:47.600583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:47.600595 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:47.600621 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:47.600634 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:47.600744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:47.601361 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 10:48:47.601397 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:48:47.602753 24567 catalog_manager.cc:1347] Generated new cluster ID: 35ea805b45ee42e2a7fe6f5afd5e3d3b
I20260924 10:48:47.602854 24567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:47.610066 24567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:47.611001 24567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:47.623140 24567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d248cba0784fb4aa3548f2f4858a91: Generated new TSK 0
I20260924 10:48:47.623936 24567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:47.635159 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24472
W20260924 10:48:47.640341 24457 connection.cc:535] client connection to 127.2.182.62:42089 recv error: Network error: failed to read from TLS socket (remote: 127.2.182.62:42089): Connection reset by peer (error 104)
2026-09-24T10:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:47 dist-test-slave-6lwt krb5kdc[24571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:47 dist-test-slave-6lwt krb5kdc[24571](info): set up 2 sockets
Sep 24 10:48:47 dist-test-slave-6lwt krb5kdc[24571](info): commencing operation
krb5kdc: starting...
W20260924 10:48:48.653399 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:48 dist-test-slave-6lwt krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:48.764806 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:33531
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:48.870211 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:48.870455 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:48.870496 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:48.874459 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:48.874511 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:48.874529 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:48.874572 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:48.874605 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:48.879729 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:33531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:48.880682 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:48.881690 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:48.889474 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:48.889521 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:48.889530 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:48.889581 24584 server_base.cc:1034] running on GCE node
I20260924 10:48:48.890182 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:48.891301 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:48.892464 24584 hybrid_clock.cc:648] HybridClock initialized: now 1790246928892434 us; error 42 us; skew 500 ppm
Sep 24 10:48:48 dist-test-slave-6lwt krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:48.895114 24584 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:48.895987 24584 webserver.cc:460] Webserver started at http://127.2.182.62:36357/ using document root <none> and password file <none>
I20260924 10:48:48.896514 24584 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:48.896603 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:48.896822 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:48.898756 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "e053faf898cc4f3293a98e3cf97ef9e3"
format_stamp: "Formatted at 2026-09-24 10:48:48 on dist-test-slave-6lwt"
server_key: "5aa503020f7441b690e656c6797e0d0f"
server_key_iv: "434ead7613cf700cbd775e8986c9adf5"
server_key_version: "encryptionkey@0"
I20260924 10:48:48.899309 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "e053faf898cc4f3293a98e3cf97ef9e3"
format_stamp: "Formatted at 2026-09-24 10:48:48 on dist-test-slave-6lwt"
server_key: "5aa503020f7441b690e656c6797e0d0f"
server_key_iv: "434ead7613cf700cbd775e8986c9adf5"
server_key_version: "encryptionkey@0"
I20260924 10:48:48.902810 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:48.905063 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:48.906142 24584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:48.906255 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "e053faf898cc4f3293a98e3cf97ef9e3"
format_stamp: "Formatted at 2026-09-24 10:48:48 on dist-test-slave-6lwt"
server_key: "5aa503020f7441b690e656c6797e0d0f"
server_key_iv: "434ead7613cf700cbd775e8986c9adf5"
server_key_version: "encryptionkey@0"
I20260924 10:48:48.906328 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:48.910665 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:48.913270 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:48.913398 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:48.919863 24584 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:33531
I20260924 10:48:48.919874 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:33531 every 8 connection(s)
I20260924 10:48:48.920728 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:48.923285 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:48.928450 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3: Bootstrap starting.
I20260924 10:48:48.929950 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24584
I20260924 10:48:48.930080 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:48.930375 2776 external_mini_cluster.cc:1428] Setting key 708f2928255e6b9cbacc7cec53542725
I20260924 10:48:48.930950 24652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:48.931739 24652 log.cc:824] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:48.933840 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3: No bootstrap required, opened a new log
I20260924 10:48:48.935738 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:48.931500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60265 (local address 127.2.182.62:33531)
0924 10:48:48.931912 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:48.931935 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:48.931966 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:48:48.932749 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:48.932766 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:48.933081 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:48.933817 (+ 736us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:48.934621 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:48.934625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:48.934635 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:48.934642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:48.934686 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:48.934702 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:48.934777 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:48.934944 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 10:48:48.935094 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260924 10:48:48.937736 24652 raft_consensus.cc:348] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } }
I20260924 10:48:48.937992 24652 raft_consensus.cc:374] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:48.938050 24652 raft_consensus.cc:729] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e053faf898cc4f3293a98e3cf97ef9e3, State: Initialized, Role: FOLLOWER
I20260924 10:48:48.938531 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:48.938505 24652 consensus_queue.cc:260] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } }
I20260924 10:48:48.938673 24652 raft_consensus.cc:388] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:48.938753 24652 raft_consensus.cc:482] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:48.938845 24652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:48.939702 24652 raft_consensus.cc:504] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } }
I20260924 10:48:48.940012 24652 leader_election.cc:302] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e053faf898cc4f3293a98e3cf97ef9e3; no voters:
I20260924 10:48:48.940357 24652 leader_election.cc:288] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:48.940445 24661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:48.940686 24661 raft_consensus.cc:686] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [term 1 LEADER]: Becoming Leader. State: Replica: e053faf898cc4f3293a98e3cf97ef9e3, State: Running, Role: LEADER
I20260924 10:48:48.941030 24661 consensus_queue.cc:237] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } }
I20260924 10:48:48.941591 24652 sys_catalog.cc:564] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:48.942013 24661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e053faf898cc4f3293a98e3cf97ef9e3. Latest consensus state: current_term: 1 leader_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } } }
I20260924 10:48:48.942184 24661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:48.942497 24662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e053faf898cc4f3293a98e3cf97ef9e3" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 33531 } } }
I20260924 10:48:48.942575 24667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:48.942649 24662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:48.945322 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:48.940527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53333 (local address 127.2.182.62:33531)
0924 10:48:48.940635 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:48.940638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:48.943862 (+ 3224us) server_negotiation.cc:407] Connection header received
0924 10:48:48.944019 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:48.944022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:48.944057 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:48.944130 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:48.944917 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:48.944920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:48.944923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:48.944933 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:48.944957 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:48.944968 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:48.945064 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:48.945157 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:48:48.945192 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:48:48.945608 24667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:48.950752 24667 catalog_manager.cc:1347] Generated new cluster ID: 7a6b8305f6e942928776f6a4fe614ace
I20260924 10:48:48.950821 24667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:48.964300 24667 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:48.964579 24667 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:48.970898 24667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e053faf898cc4f3293a98e3cf97ef9e3: Generated new TSK 0
I20260924 10:48:48.971487 24667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:48.977165 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24584
2026-09-24T10:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:49 dist-test-slave-6lwt krb5kdc[24683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:49 dist-test-slave-6lwt krb5kdc[24683](info): set up 2 sockets
Sep 24 10:48:49 dist-test-slave-6lwt krb5kdc[24683](info): commencing operation
krb5kdc: starting...
W20260924 10:48:50.009704 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:50.125873 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:40415
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:40415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:50.230948 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:50.231168 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:50.231202 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:50.234891 24696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:50.234959 24696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:50.234977 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:50.234993 24696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:50.235008 24696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:50.239787 24696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:40415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:50.240697 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:50.241676 24696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:50.249606 24696 server_base.cc:1034] running on GCE node
W20260924 10:48:50.249562 24701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:50.249562 24702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:50.249541 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:50.250305 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:50.251298 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:50.252494 24696 hybrid_clock.cc:648] HybridClock initialized: now 1790246930252483 us; error 29 us; skew 500 ppm
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246930, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:50.263317 24696 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:50.264166 24696 webserver.cc:460] Webserver started at http://127.2.182.62:36587/ using document root <none> and password file <none>
I20260924 10:48:50.264711 24696 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:50.264792 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:50.265048 24696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:50.266933 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "a2abbff82e944d7392bd9b0cfec6a307"
format_stamp: "Formatted at 2026-09-24 10:48:50 on dist-test-slave-6lwt"
server_key: "8577e5fb3ccea6dc8a62cac0d75f1c72"
server_key_iv: "d51c30bf5b1aba5bc7ed582418e4779c"
server_key_version: "encryptionkey@0"
I20260924 10:48:50.267462 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "a2abbff82e944d7392bd9b0cfec6a307"
format_stamp: "Formatted at 2026-09-24 10:48:50 on dist-test-slave-6lwt"
server_key: "8577e5fb3ccea6dc8a62cac0d75f1c72"
server_key_iv: "d51c30bf5b1aba5bc7ed582418e4779c"
server_key_version: "encryptionkey@0"
I20260924 10:48:50.270880 24696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:50.273069 24711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:50.274165 24696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:50.274303 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "a2abbff82e944d7392bd9b0cfec6a307"
format_stamp: "Formatted at 2026-09-24 10:48:50 on dist-test-slave-6lwt"
server_key: "8577e5fb3ccea6dc8a62cac0d75f1c72"
server_key_iv: "d51c30bf5b1aba5bc7ed582418e4779c"
server_key_version: "encryptionkey@0"
I20260924 10:48:50.274416 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:50.291903 24696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:50.294955 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:50.295145 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:50.301919 24696 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:40415
I20260924 10:48:50.301937 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:40415 every 8 connection(s)
I20260924 10:48:50.302810 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:50.305449 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:50.310786 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307: Bootstrap starting.
I20260924 10:48:50.312055 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24696
I20260924 10:48:50.312191 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:50.312460 2776 external_mini_cluster.cc:1428] Setting key af5dcfd116e48cf6a048e0eafd753658
I20260924 10:48:50.313642 24764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:50.314505 24764 log.cc:824] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:50.316533 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307: No bootstrap required, opened a new log
I20260924 10:48:50.320246 24764 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } }
I20260924 10:48:50.320437 24764 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:50.320477 24764 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2abbff82e944d7392bd9b0cfec6a307, State: Initialized, Role: FOLLOWER
I20260924 10:48:50.320863 24764 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } }
I20260924 10:48:50.321034 24764 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:50.321115 24764 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:50.321205 24764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:50.322094 24764 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } }
I20260924 10:48:50.322440 24764 leader_election.cc:302] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2abbff82e944d7392bd9b0cfec6a307; no voters:
I20260924 10:48:50.322710 24764 leader_election.cc:288] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:50.322820 24769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:50.323035 24769 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [term 1 LEADER]: Becoming Leader. State: Replica: a2abbff82e944d7392bd9b0cfec6a307, State: Running, Role: LEADER
I20260924 10:48:50.323380 24769 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } }
I20260924 10:48:50.323908 24764 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:50.325114 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2abbff82e944d7392bd9b0cfec6a307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } } }
I20260924 10:48:50.325116 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2abbff82e944d7392bd9b0cfec6a307. Latest consensus state: current_term: 1 leader_uuid: "a2abbff82e944d7392bd9b0cfec6a307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2abbff82e944d7392bd9b0cfec6a307" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 40415 } } }
I20260924 10:48:50.325253 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:50.325253 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:50.325668 24779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:50.327209 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:50.313545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47051 (local address 127.2.182.62:40415)
0924 10:48:50.313972 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:50.313989 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:50.314028 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:48:50.314865 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:50.314887 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:50.315238 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:50.315569 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:50.316173 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:50.317569 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:50.318360 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:50.319104 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:50.319291 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:50.321744 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:50.321748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:50.321759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:50.321783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:50.323696 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:50.324233 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:50.324240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:50.324248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:50.324377 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:50.324709 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:50.324712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:50.324714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:50.325234 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:50.326128 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:50.326294 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:48:50.326503 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260924 10:48:50.329025 24779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:50.329669 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:50.334394 24779 catalog_manager.cc:1347] Generated new cluster ID: c2aa9052acf5498baaa78002bf46c11c
I20260924 10:48:50.334448 24779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:50.338461 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:50.331526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49471 (local address 127.2.182.62:40415)
0924 10:48:50.331621 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:50.331625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:50.331755 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:48:50.331841 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:50.331844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:50.331882 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:50.331954 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:50.332544 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:50.333201 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:50.333996 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:50.334241 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:50.334355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:50.335549 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:50.335552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:50.335554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:50.335577 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:50.336864 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:50.337291 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:50.337296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:50.337298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:50.337383 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:50.337630 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:50.337634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:50.337636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:50.337792 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:50.337943 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:50.338174 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 10:48:50.338278 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:50.342029 24779 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:50.342979 24779 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:50.347782 24779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2abbff82e944d7392bd9b0cfec6a307: Generated new TSK 0
I20260924 10:48:50.348356 24779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:50.356693 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24696
2026-09-24T10:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24795](info): set up 2 sockets
Sep 24 10:48:50 dist-test-slave-6lwt krb5kdc[24795](info): commencing operation
krb5kdc: starting...
W20260924 10:48:51.377410 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:51 dist-test-slave-6lwt krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:51.495620 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41347
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:51.599249 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:51.599475 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:51.599514 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:51.603471 24808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:51.603525 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:51.603544 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:51.603560 24808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:51.603621 24808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:51.608712 24808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:51.609740 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:51.610644 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:51.618739 24814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:51.618841 24808 server_base.cc:1034] running on GCE node
W20260924 10:48:51.618798 24816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:51.618808 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:51.619503 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:51.620553 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:51.621726 24808 hybrid_clock.cc:648] HybridClock initialized: now 1790246931621713 us; error 24 us; skew 500 ppm
Sep 24 10:48:51 dist-test-slave-6lwt krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:51.624261 24808 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:51.625167 24808 webserver.cc:460] Webserver started at http://127.2.182.62:33829/ using document root <none> and password file <none>
I20260924 10:48:51.625710 24808 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:51.625793 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:51.626024 24808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:51.627954 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "3dfc627f5cb44681869d6edf1e4540f0"
format_stamp: "Formatted at 2026-09-24 10:48:51 on dist-test-slave-6lwt"
server_key: "988b8f628eb6ef0d71f536c8b3fded14"
server_key_iv: "76a9691d039b767953959f82e9d3f505"
server_key_version: "encryptionkey@0"
I20260924 10:48:51.628484 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "3dfc627f5cb44681869d6edf1e4540f0"
format_stamp: "Formatted at 2026-09-24 10:48:51 on dist-test-slave-6lwt"
server_key: "988b8f628eb6ef0d71f536c8b3fded14"
server_key_iv: "76a9691d039b767953959f82e9d3f505"
server_key_version: "encryptionkey@0"
I20260924 10:48:51.631812 24808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:48:51.634039 24823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:51.635108 24808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:48:51.635239 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "3dfc627f5cb44681869d6edf1e4540f0"
format_stamp: "Formatted at 2026-09-24 10:48:51 on dist-test-slave-6lwt"
server_key: "988b8f628eb6ef0d71f536c8b3fded14"
server_key_iv: "76a9691d039b767953959f82e9d3f505"
server_key_version: "encryptionkey@0"
I20260924 10:48:51.635344 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:51.649336 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:51.651772 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:51.651959 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:51.658939 24808 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:41347
I20260924 10:48:51.658967 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:41347 every 8 connection(s)
I20260924 10:48:51.659857 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:51.660848 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24808
I20260924 10:48:51.661038 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:51.661330 2776 external_mini_cluster.cc:1428] Setting key b2a1a548a49cc5275bdf1ce299d7c73e
I20260924 10:48:51.662799 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:51.669436 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0: Bootstrap starting.
I20260924 10:48:51.670835 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:51.662690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43219 (local address 127.2.182.62:41347)
0924 10:48:51.663113 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:51.663131 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:51.663172 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:48:51.663983 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:51.664004 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:51.664412 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:51.664738 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:51.665152 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:51.666566 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:51.667388 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:51.668079 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:51.668235 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:51.669008 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:51.669014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:51.669029 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:51.669476 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:51.669539 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:51.669568 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:51.669658 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:51.669765 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 10:48:51.669960 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":286,"thread_start_us":151,"threads_started":1}
I20260924 10:48:51.671968 24876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:51.672751 24876 log.cc:824] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:51.673564 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:51.674634 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0: No bootstrap required, opened a new log
I20260924 10:48:51.678025 24876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } }
I20260924 10:48:51.678292 24876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:51.678376 24876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dfc627f5cb44681869d6edf1e4540f0, State: Initialized, Role: FOLLOWER
I20260924 10:48:51.678838 24876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } }
I20260924 10:48:51.678956 24876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:51.679045 24876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:51.679143 24876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:51.679888 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:51.675461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51973 (local address 127.2.182.62:41347)
0924 10:48:51.675573 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:51.675577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:51.675714 (+ 137us) server_negotiation.cc:407] Connection header received
0924 10:48:51.675843 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:51.675844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:51.675877 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:51.675937 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:51.676175 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:51.677207 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:51.678244 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:51.678578 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:51.678704 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:51.679493 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:51.679497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:51.679499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:51.679511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:51.679539 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:51.679550 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:51.679621 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:51.679708 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:48:51.679766 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:51.680135 24876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } }
I20260924 10:48:51.680469 24876 leader_election.cc:302] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dfc627f5cb44681869d6edf1e4540f0; no voters:
I20260924 10:48:51.680751 24876 leader_election.cc:288] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:51.680821 24886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:51.681123 24886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [term 1 LEADER]: Becoming Leader. State: Replica: 3dfc627f5cb44681869d6edf1e4540f0, State: Running, Role: LEADER
I20260924 10:48:51.681466 24886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } }
I20260924 10:48:51.681908 24876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:51.683048 24887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dfc627f5cb44681869d6edf1e4540f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } } }
I20260924 10:48:51.683174 24887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:51.683497 24888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dfc627f5cb44681869d6edf1e4540f0. Latest consensus state: current_term: 1 leader_uuid: "3dfc627f5cb44681869d6edf1e4540f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dfc627f5cb44681869d6edf1e4540f0" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41347 } } }
I20260924 10:48:51.683552 24895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:51.683657 24888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:51.686574 24895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:51.691622 24895 catalog_manager.cc:1347] Generated new cluster ID: 7d5f1b8f7ef04a82bf21c39bc3ef1e75
I20260924 10:48:51.691681 24895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:51.722990 24895 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:51.723908 24895 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:51.728281 24895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dfc627f5cb44681869d6edf1e4540f0: Generated new TSK 0
I20260924 10:48:51.729064 24895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:51.743167 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24808
2026-09-24T10:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:51 dist-test-slave-6lwt krb5kdc[24907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:51 dist-test-slave-6lwt krb5kdc[24907](info): set up 2 sockets
Sep 24 10:48:51 dist-test-slave-6lwt krb5kdc[24907](info): commencing operation
krb5kdc: starting...
W20260924 10:48:52.757328 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:52 dist-test-slave-6lwt krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:52.875799 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41843
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:52.981396 24920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:52.981614 24920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:52.981649 24920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:52.985479 24920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:52.985523 24920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:52.985541 24920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:52.985558 24920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:52.985571 24920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:52.990396 24920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:41843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:52.991247 24920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:52.992125 24920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:53.000027 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:53.000029 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:53.000064 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:53.000188 24920 server_base.cc:1034] running on GCE node
I20260924 10:48:53.000950 24920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:53.002010 24920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:53.003163 24920 hybrid_clock.cc:648] HybridClock initialized: now 1790246933003138 us; error 43 us; skew 500 ppm
Sep 24 10:48:53 dist-test-slave-6lwt krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246933, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:53.005843 24920 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:53.006644 24920 webserver.cc:460] Webserver started at http://127.2.182.62:39941/ using document root <none> and password file <none>
I20260924 10:48:53.007161 24920 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:53.007223 24920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:53.007459 24920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:53.009457 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "478c7e42d8f24677b0cd658c20cadbac"
format_stamp: "Formatted at 2026-09-24 10:48:53 on dist-test-slave-6lwt"
server_key: "8ba5c79f2a90d70b3061df07d7d50fee"
server_key_iv: "ddafdb74fceb4e6c6e64ba640c2ab913"
server_key_version: "encryptionkey@0"
I20260924 10:48:53.010061 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "478c7e42d8f24677b0cd658c20cadbac"
format_stamp: "Formatted at 2026-09-24 10:48:53 on dist-test-slave-6lwt"
server_key: "8ba5c79f2a90d70b3061df07d7d50fee"
server_key_iv: "ddafdb74fceb4e6c6e64ba640c2ab913"
server_key_version: "encryptionkey@0"
I20260924 10:48:53.013643 24920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:48:53.016076 24935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:53.017328 24920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 10:48:53.017464 24920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "478c7e42d8f24677b0cd658c20cadbac"
format_stamp: "Formatted at 2026-09-24 10:48:53 on dist-test-slave-6lwt"
server_key: "8ba5c79f2a90d70b3061df07d7d50fee"
server_key_iv: "ddafdb74fceb4e6c6e64ba640c2ab913"
server_key_version: "encryptionkey@0"
I20260924 10:48:53.017567 24920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:53.022017 24920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:53.024709 24920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:53.024930 24920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:53.031553 24920 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:41843
I20260924 10:48:53.031564 24987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:41843 every 8 connection(s)
I20260924 10:48:53.032446 24920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:53.035140 24988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:53.040416 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac: Bootstrap starting.
I20260924 10:48:53.041002 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 24920
I20260924 10:48:53.041149 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:53.041417 2776 external_mini_cluster.cc:1428] Setting key a18fedb500bafd211a4bf52dfdff25c4
I20260924 10:48:53.043263 24988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:53.044155 24988 log.cc:824] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:53.046393 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac: No bootstrap required, opened a new log
I20260924 10:48:53.047055 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:53.042654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45671 (local address 127.2.182.62:41843)
0924 10:48:53.043181 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:53.043204 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:53.043253 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:48:53.044138 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:53.044200 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:53.044554 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:53.044853 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:53.045636 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:53.045642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:53.045657 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:53.045671 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:53.045735 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:53.045763 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:53.045886 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:53.045965 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:48:53.046108 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":101,"threads_started":1}
I20260924 10:48:53.049793 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:53.050628 24988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } }
I20260924 10:48:53.050872 24988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:48:53.050966 24988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478c7e42d8f24677b0cd658c20cadbac, State: Initialized, Role: FOLLOWER
I20260924 10:48:53.051441 24988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } }
I20260924 10:48:53.051597 24988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:48:53.051733 24988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:48:53.051975 24988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:53.053062 24988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } }
I20260924 10:48:53.053450 24988 leader_election.cc:302] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 478c7e42d8f24677b0cd658c20cadbac; no voters:
I20260924 10:48:53.053601 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:53.051853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58431 (local address 127.2.182.62:41843)
0924 10:48:53.052004 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:53.052008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:53.052139 (+ 131us) server_negotiation.cc:407] Connection header received
0924 10:48:53.052272 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:53.052274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:53.052311 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:53.052399 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:53.053103 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:53.053106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:53.053109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:48:53.053120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:53.053147 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:48:53.053161 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:53.053259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:53.053417 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:48:53.053452 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:48:53.053817 24988 leader_election.cc:288] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:48:53.053898 24998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:53.054093 24998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [term 1 LEADER]: Becoming Leader. State: Replica: 478c7e42d8f24677b0cd658c20cadbac, State: Running, Role: LEADER
I20260924 10:48:53.054452 24998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } }
I20260924 10:48:53.054989 24988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:53.056119 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478c7e42d8f24677b0cd658c20cadbac. Latest consensus state: current_term: 1 leader_uuid: "478c7e42d8f24677b0cd658c20cadbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } } }
I20260924 10:48:53.056252 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:53.056658 25007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:48:53.056974 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "478c7e42d8f24677b0cd658c20cadbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478c7e42d8f24677b0cd658c20cadbac" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41843 } } }
I20260924 10:48:53.057083 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:53.059608 25007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:53.064489 25007 catalog_manager.cc:1347] Generated new cluster ID: 8693f41dfcde4265a019b3c2acb4e469
I20260924 10:48:53.064551 25007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:53.080281 25007 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:53.080606 25007 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:53.089620 25007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 478c7e42d8f24677b0cd658c20cadbac: Generated new TSK 0
I20260924 10:48:53.090396 25007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:48:53.094478 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 24920
2026-09-24T10:48:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1350 ms)
[----------] 18 tests from AuthTokenIssuingTest (24705 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:53 dist-test-slave-6lwt krb5kdc[25019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:53 dist-test-slave-6lwt krb5kdc[25019](info): set up 2 sockets
Sep 24 10:48:53 dist-test-slave-6lwt krb5kdc[25019](info): commencing operation
krb5kdc: starting...
W20260924 10:48:54.113580 2776 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:54.225366 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:45603
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:54.329141 25032 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:54.329430 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:54.329516 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:54.329594 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:54.333406 25032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:54.333477 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:54.333516 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:54.333564 25032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:54.333603 25032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:54.338668 25032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:54.339625 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:54.340543 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:54.348227 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:54.348274 25032 server_base.cc:1034] running on GCE node
W20260924 10:48:54.348301 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:54.348507 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:54.349100 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:54.349970 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:54.351116 25032 hybrid_clock.cc:648] HybridClock initialized: now 1790246934351087 us; error 39 us; skew 500 ppm
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:54.353856 25032 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:54.354643 25032 webserver.cc:460] Webserver started at http://127.2.182.62:43541/ using document root <none> and password file <none>
I20260924 10:48:54.355098 25032 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:54.355149 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:54.355293 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:54.357121 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "455f9158727844e190aba64f18a910fd"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "a6531a85ff7d4b1535c0a0a04ae6933a"
server_key_iv: "81181ba0f32022e770b3ae0e06d3a4ec"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.357626 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "455f9158727844e190aba64f18a910fd"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "a6531a85ff7d4b1535c0a0a04ae6933a"
server_key_iv: "81181ba0f32022e770b3ae0e06d3a4ec"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.360841 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:54.363037 25047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:54.364143 25032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:48:54.364235 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "455f9158727844e190aba64f18a910fd"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "a6531a85ff7d4b1535c0a0a04ae6933a"
server_key_iv: "81181ba0f32022e770b3ae0e06d3a4ec"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.364310 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:54.389277 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:54.392320 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:54.392459 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:54.398648 25032 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:45603
I20260924 10:48:54.398904 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:45603 every 8 connection(s)
I20260924 10:48:54.399524 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:54.401127 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25032
I20260924 10:48:54.401252 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:54.401511 2776 external_mini_cluster.cc:1428] Setting key 8c7930afd557613f1fea8a8a60ccb910
I20260924 10:48:54.402155 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:54.405759 25100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.182.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.61@KRBTEST.COM" created.
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:54.420492 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.407974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:45603 (local address 127.0.0.1:53474)
0924 10:48:54.408446 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.408496 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.409389 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.410064 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.410068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.410344 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.410564 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.410575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.411810 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.411813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.412530 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.412557 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.413421 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.413424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.413504 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.414029 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.414043 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.415749 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.418054 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.418061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.418068 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.418260 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.418612 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.418615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.418616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.418719 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.419368 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.419372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.419470 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.419591 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.419811 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260924 10:48:54.420499 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.408932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.2.182.62:45603)
0924 10:48:54.409206 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.409215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.409259 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:48:54.409665 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.409669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.409832 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.410057 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.410693 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.411698 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.412671 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.413281 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.413458 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.415946 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.415952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.415961 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.415980 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.417955 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.418367 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.418373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.418375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.418506 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.418809 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.418815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.418818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.419266 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.419403 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.419728 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.419897 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":8576,"thread_start_us":74,"threads_started":1}
I20260924 10:48:54.422717 25100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:54.423360 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.423149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39126)
0924 10:48:54.423244 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.423304 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 10:48:54.423614 25048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
I20260924 10:48:54.424288 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.424122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39136)
0924 10:48:54.424212 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.424249 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:48:54.424491 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:41421: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
I20260924 10:48:54.428234 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:41421
--webserver_interface=127.2.182.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:54.473770 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } attempt: 1
I20260924 10:48:54.474557 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.474285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39152)
0924 10:48:54.474421 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.474484 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:48:54.475330 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.475090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39156)
0924 10:48:54.475203 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.475274 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:48:54.475490 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:41421: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
W20260924 10:48:54.533429 25107 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:54.533659 25107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:54.533703 25107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:54.533803 25107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:54.537528 25107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:54.537571 25107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:54.537590 25107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:54.537642 25107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:54.537669 25107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:54.542577 25107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:54.543500 25107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:54.543632 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } attempt: 2
I20260924 10:48:54.544437 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.544152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39160)
0924 10:48:54.544307 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.544367 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:48:54.544703 25107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:54.545490 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.545282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39176)
0924 10:48:54.545383 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.545434 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:48:54.545636 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:41421: Network error: Client connection negotiation failed: client connection to 127.2.182.61:41421: connect: Connection refused (error 111)
W20260924 10:48:54.552845 25113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:54.552867 25112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:54.553028 25107 server_base.cc:1034] running on GCE node
W20260924 10:48:54.552922 25115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:54.553663 25107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:54.554728 25107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:54.555886 25107 hybrid_clock.cc:648] HybridClock initialized: now 1790246934555854 us; error 41 us; skew 500 ppm
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:54.558785 25107 init.cc:377] Logged in from keytab as kudu/127.2.182.61@KRBTEST.COM (short username kudu)
I20260924 10:48:54.559621 25107 webserver.cc:460] Webserver started at http://127.2.182.61:35493/ using document root <none> and password file <none>
I20260924 10:48:54.560230 25107 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:54.560323 25107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:54.560552 25107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:54.562669 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data/instance:
uuid: "31473d133490415fa07f599169486a74"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "5be953606828a36ee6533f76de06d953"
server_key_iv: "c1645b6e181693e75b28df27c1c53b60"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.563249 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal/instance:
uuid: "31473d133490415fa07f599169486a74"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "5be953606828a36ee6533f76de06d953"
server_key_iv: "c1645b6e181693e75b28df27c1c53b60"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.567040 25107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:48:54.569566 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:54.570719 25107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:54.570865 25107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "31473d133490415fa07f599169486a74"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "5be953606828a36ee6533f76de06d953"
server_key_iv: "c1645b6e181693e75b28df27c1c53b60"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.570979 25107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:54.583869 25107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:54.587059 25107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:54.587280 25107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:54.593724 25107 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.61:41421
I20260924 10:48:54.593983 25174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.61:41421 every 8 connection(s)
I20260924 10:48:54.594694 25107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
I20260924 10:48:54.597259 25175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:54.600553 25175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:54.603271 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25107
I20260924 10:48:54.603391 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-1/wal/instance
I20260924 10:48:54.603677 2776 external_mini_cluster.cc:1428] Setting key 71c3794a42028944cc79155cf42cf379
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.60@KRBTEST.COM; defaulting to no policy
I20260924 10:48:54.615037 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.602234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.2.182.62:45603)
0924 10:48:54.602374 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.602378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.603180 (+ 802us) server_negotiation.cc:407] Connection header received
0924 10:48:54.604327 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.604330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.604407 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.604517 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.605802 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.606377 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.608311 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.608541 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.608624 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.611447 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.611450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.611452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.611473 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.613058 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.613531 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.613536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.613538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.613618 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.613980 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.613982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.613984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.614145 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.614242 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.614700 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.614884 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.60@KRBTEST.COM" created.
I20260924 10:48:54.615796 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.602563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:45603 (local address 127.0.0.1:53482)
0924 10:48:54.603030 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.603072 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.604075 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.604703 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.604711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.605131 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.605645 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.605658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.606522 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.606529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.608130 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.608144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.608699 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.608704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.608975 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.609618 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.609645 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.611278 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.613164 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.613170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.613183 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.613431 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.613714 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.613717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.613719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.613875 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.614243 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.614250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.614464 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.614646 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.614953 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260924 10:48:54.617024 25175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
I20260924 10:48:54.626161 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.617520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39178)
0924 10:48:54.617852 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.617878 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.617950 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.618296 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.618298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.618452 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.618528 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.618533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.619611 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.619617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.620127 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.620134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.620667 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.620670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.620758 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.621261 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.621271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.622450 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.624371 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.624374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.624375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.624628 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.624978 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.624985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.624987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.625067 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.625600 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.625602 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.625679 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.625806 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.625944 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260924 10:48:54.626251 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.617590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.2.182.61:41421)
0924 10:48:54.617844 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.617854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.617925 (+ 71us) server_negotiation.cc:407] Connection header received
0924 10:48:54.618037 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.618042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.618185 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.618337 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.618633 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.619506 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.620233 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.620566 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.620671 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.622616 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.622623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.622637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.622656 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.624273 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.624778 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.624782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.624783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.624854 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.625148 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.625150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.625151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.625484 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.625626 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.625902 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.626066 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260924 10:48:54.627933 25175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:54.628579 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.628369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36340)
0924 10:48:54.628473 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.628533 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:48:54.628757 25123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
I20260924 10:48:54.629607 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.629453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36346)
0924 10:48:54.629526 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.629565 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 10:48:54.629765 25175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
I20260924 10:48:54.631547 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:43453
--webserver_interface=127.2.182.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:54.636837 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } attempt: 3
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
I20260924 10:48:54.645831 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.637365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.2.182.61:41421)
0924 10:48:54.637507 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.637513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.637530 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:48:54.637570 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.637575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.637627 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.637725 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.638102 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.639043 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.640178 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.640675 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.640806 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.642969 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.642972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.642974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.642994 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.644416 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.644846 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.644849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.644850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.644953 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.645212 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.645214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.645215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.645348 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.645424 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.645668 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.645746 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:54.645980 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.637295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39190)
0924 10:48:54.637391 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.637406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.637492 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.637733 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.637735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.637894 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.637986 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.638002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.639152 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.639154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.640041 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.640049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.640788 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.640790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.640873 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.641425 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.641435 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.642834 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.644513 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.644516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.644519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.644761 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.645030 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.645035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.645037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.645125 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.645436 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.645439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.645532 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.645633 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.645760 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 10:48:54.646837 25100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:54.647465 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.647259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36362)
0924 10:48:54.647369 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.647413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:48:54.648114 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.647899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36366)
0924 10:48:54.648029 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.648068 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 10:48:54.648286 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
I20260924 10:48:54.679018 25175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 1
I20260924 10:48:54.679427 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 1
I20260924 10:48:54.679750 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.679468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36368)
0924 10:48:54.679611 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.679676 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 10:48:54.680003 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.679750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36372)
0924 10:48:54.679881 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.679934 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:48:54.680630 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.680400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36388)
0924 10:48:54.680524 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.680570 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:48:54.680774 25175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
I20260924 10:48:54.680941 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.680561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36392)
0924 10:48:54.680677 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.680847 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:48:54.681113 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
W20260924 10:48:54.744624 25182 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:54.744977 25182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:54.745050 25182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:54.745131 25182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:48:54.748945 25175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 2
W20260924 10:48:54.749049 25182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:54.749114 25182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:54.749171 25182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:54.749212 25182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:54.749233 25182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:54.749703 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.749437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36400)
0924 10:48:54.749573 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.749629 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:48:54.750404 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.750227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36414)
0924 10:48:54.750309 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.750348 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 10:48:54.750586 25175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
I20260924 10:48:54.754663 25182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:45603,127.2.182.61:41421,127.2.182.60:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:43453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:54.755650 25182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:54.756264 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 2
I20260924 10:48:54.756600 25182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:54.756873 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.756632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36428)
0924 10:48:54.756749 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.756804 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:48:54.757570 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.757373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36436)
0924 10:48:54.757472 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.757514 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 10:48:54.757688 25100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:43453: Network error: Client connection negotiation failed: client connection to 127.2.182.60:43453: connect: Connection refused (error 111)
W20260924 10:48:54.764598 25190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:54.764621 25188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:54.764622 25187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:54.764808 25182 server_base.cc:1034] running on GCE node
I20260924 10:48:54.765426 25182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:54.766517 25182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:54.767684 25182 hybrid_clock.cc:648] HybridClock initialized: now 1790246934767647 us; error 45 us; skew 500 ppm
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:54.770280 25182 init.cc:377] Logged in from keytab as kudu/127.2.182.60@KRBTEST.COM (short username kudu)
I20260924 10:48:54.771045 25182 webserver.cc:460] Webserver started at http://127.2.182.60:43987/ using document root <none> and password file <none>
I20260924 10:48:54.771565 25182 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:54.771668 25182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:54.771876 25182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:54.773864 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data/instance:
uuid: "c4a8514b274f4263a6ff245798d5d546"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "4f90e92ee46659bb1b9de176ed5597f8"
server_key_iv: "92c707b3b0e553100ab01a3e6df65947"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.774439 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal/instance:
uuid: "c4a8514b274f4263a6ff245798d5d546"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "4f90e92ee46659bb1b9de176ed5597f8"
server_key_iv: "92c707b3b0e553100ab01a3e6df65947"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.777997 25182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:54.780184 25197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:54.781322 25182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:54.781459 25182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "c4a8514b274f4263a6ff245798d5d546"
format_stamp: "Formatted at 2026-09-24 10:48:54 on dist-test-slave-6lwt"
server_key: "4f90e92ee46659bb1b9de176ed5597f8"
server_key_iv: "92c707b3b0e553100ab01a3e6df65947"
server_key_version: "encryptionkey@0"
I20260924 10:48:54.781556 25182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:54.793087 25182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:54.795583 25182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:54.795756 25182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:54.801892 25182 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.60:43453
I20260924 10:48:54.802135 25249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.60:43453 every 8 connection(s)
I20260924 10:48:54.802762 25182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
I20260924 10:48:54.805258 25250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:54.806454 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25182
I20260924 10:48:54.806587 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246662595599-2776-0/minicluster-data/master-2/wal/instance
I20260924 10:48:54.806840 2776 external_mini_cluster.cc:1428] Setting key 65bac304ce4c739131b7cb5cc77fbdd2
I20260924 10:48:54.809535 25250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:54.820868 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.812047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.2.182.62:45603)
0924 10:48:54.812170 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.812173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.812184 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:48:54.812238 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.812243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.812287 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.812373 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.813078 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.813589 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.814560 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.814949 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.815062 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.817322 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.817327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.817329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.817360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.818885 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.819385 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.819389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.819390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.819465 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.819769 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.819772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.819773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.819962 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.820052 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.820429 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.820560 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:48:54.821466 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.811222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:45603 (local address 127.0.0.1:53494)
0924 10:48:54.811569 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.811609 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.811700 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.812420 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.812424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.812780 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.812956 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.812964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.813734 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.813737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.814451 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.814459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.815088 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.815091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.815180 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.815749 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.815767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.817205 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.818977 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.818986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.818994 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.819293 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.819545 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.819548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.819550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.819669 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.820052 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.820056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.820181 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.820369 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.820659 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":26,"thread_start_us":133,"threads_started":1}
I20260924 10:48:54.821465 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.808131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.2.182.60:43453)
0924 10:48:54.808601 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.808643 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.808683 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:48:54.809524 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.809546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.809917 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.810251 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.811086 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.812133 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.813131 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.813568 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.813729 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.816709 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.816716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.816726 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.816749 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.818759 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.819258 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.819263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.819265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.819423 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.819702 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.819704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.819705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.820113 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.820291 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.820546 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.820712 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":2432,"thread_start_us":115,"threads_started":1}
I20260924 10:48:54.823207 25250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:54.830752 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.823605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:41421 (local address 127.0.0.1:39200)
0924 10:48:54.823701 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.823712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.823791 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.824181 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.824183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.824340 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.824424 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.824429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.825187 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.825194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.825679 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.825687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.826097 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.826101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.826191 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.826700 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.826710 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.827691 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.829395 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.829397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.829399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.829565 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.829828 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.829831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.829833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.829906 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.830311 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.830314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.830400 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.830497 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.830608 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 10:48:54.830780 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.823860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.2.182.61:41421)
0924 10:48:54.823994 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.824003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.824014 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:48:54.824047 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.824049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.824084 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.824169 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.824537 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.825089 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.825787 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.825996 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.826098 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.827813 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.827816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.827819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.827843 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.829289 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.829652 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.829656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.829657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.829731 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.829983 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.829987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.829988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.830159 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.830318 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.830548 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.830646 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:48:54.831213 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.823520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.2.182.62:45603)
0924 10:48:54.823613 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.823617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.823652 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:48:54.823784 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.823787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.823833 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.823909 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.824461 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.824976 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.825714 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.825929 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.826023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.828216 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.828219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.828221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.828246 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.829453 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.829812 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.829816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.829818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.829889 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.830146 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.830150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.830152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.830327 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.830539 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.830660 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.830753 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:48:54.831343 25250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
I20260924 10:48:54.839411 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.831849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.2.182.60:43453)
0924 10:48:54.831938 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.831941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.831951 (+ 10us) server_negotiation.cc:407] Connection header received
0924 10:48:54.831978 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.831980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.832004 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.832070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.832397 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.833046 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.834029 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.834247 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.834352 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.836191 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.836193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.836195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.836213 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.837956 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.838305 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.838310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.838312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.838404 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.838666 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.838671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.838674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.838878 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.838950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.839206 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.839296 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
I20260924 10:48:54.839448 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.831715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36462)
0924 10:48:54.831786 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.831797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.831854 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.832079 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.832082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.832216 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.832292 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.832300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.833162 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.833165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.833950 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.833960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.834348 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.834351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.834446 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.834736 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.834749 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.836082 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.838054 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.838061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.838062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.838206 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.838495 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.838500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.838504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.838587 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.838968 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.838973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.839060 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.839139 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.839259 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":17}
I20260924 10:48:54.840224 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.832251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.2.182.61:41421)
0924 10:48:54.832362 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.832366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.832378 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:48:54.832430 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.832433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.832474 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.832533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.833123 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.833863 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.834498 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.834799 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.834894 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.837202 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.837205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.837207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.837232 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.838472 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.838849 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.838852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.838853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.838938 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.839190 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.839193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.839194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.839373 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.839580 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.840005 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.840103 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:54.841733 25175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 3
I20260924 10:48:54.842617 25250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: Bootstrap starting.
I20260924 10:48:54.843119 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:54.845261 25250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:54.844434 25074 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "455f9158727844e190aba64f18a910fd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53506
I20260924 10:48:54.846086 25250 log.cc:824] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:48:54.848564 25250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: No bootstrap required, opened a new log
I20260924 10:48:54.849989 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.842362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36464)
0924 10:48:54.842478 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.842491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.842584 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.843220 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.843223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.843381 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.843444 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.843449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.844091 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.844096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.844619 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.844627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.845074 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.845077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.845154 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.845563 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.845576 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.846686 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.848497 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.848500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.848502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.848733 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.849074 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.849077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.849079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.849158 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.849546 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.849549 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.849629 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.849726 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.849845 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:48:54.850016 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.842716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.2.182.60:43453)
0924 10:48:54.843036 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.843040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.843052 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:48:54.843082 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.843086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.843127 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.843206 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.843546 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.843996 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.844725 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.844977 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.845070 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.846816 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.846819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.846820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.846838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.848375 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.848868 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.848873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.848875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.848988 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.849245 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.849248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.849250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.849454 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.849549 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.849781 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.849887 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260924 10:48:54.852169 25250 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.852403 25175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: Bootstrap starting.
I20260924 10:48:54.852422 25250 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a8514b274f4263a6ff245798d5d546, State: Initialized, Role: FOLLOWER
I20260924 10:48:54.852756 25250 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.853219 25256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.853349 25256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.854033 25250 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:54.854564 25175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:54.855306 25175 log.cc:824] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:54.856860 25100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } attempt: 3
I20260924 10:48:54.857326 25175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: No bootstrap required, opened a new log
W20260924 10:48:54.859983 25268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:54.860052 25268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:54.857687 25182 master_runner.cc:186] Error getting master registration for 127.2.182.62:45603: OK, instance_id { permanent_uuid: "455f9158727844e190aba64f18a910fd" instance_seqno: 1790246934396870 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:48:54.861029 25182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:48:54.861065 25175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.861286 25175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31473d133490415fa07f599169486a74, State: Initialized, Role: FOLLOWER
I20260924 10:48:54.861583 25175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.862046 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.862233 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.862820 25175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 10:48:54 dist-test-slave-6lwt krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
W20260924 10:48:54.869021 25281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:54.869088 25281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:54.866627 25107 master_runner.cc:186] Error getting master registration for 127.2.182.62:45603: OK, instance_id { permanent_uuid: "455f9158727844e190aba64f18a910fd" instance_seqno: 1790246934396870 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:48:54.869639 25107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:48:54.869729 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.857203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:43453 (local address 127.0.0.1:36470)
0924 10:48:54.857460 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:54.857475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:54.857542 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:54.861175 (+ 3633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:54.861178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:54.861341 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:54.861433 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.861440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.862353 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.862356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.863198 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:54.863209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.863791 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.863795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:54.863891 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.864420 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:54.864435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:54.865966 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:54.867979 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.867982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.867984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.868213 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.868570 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:54.868573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:54.868575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:54.868676 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:54.869236 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:54.869239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:54.869346 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:48:54.869454 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:48:54.869580 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 10:48:54.870404 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.857685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.2.182.60:43453)
0924 10:48:54.860983 (+ 3298us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.860987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.860999 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:48:54.861029 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.861031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.861067 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.861149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.861555 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.862229 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.863346 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.863658 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.863791 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.866109 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.866112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.866115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.866141 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.867818 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.868335 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.868338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.868340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.868451 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.868789 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.868792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.868794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.869033 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.869122 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.870138 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.870267 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3235}
I20260924 10:48:54.872565 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: Bootstrap starting.
I20260924 10:48:54.875036 25100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:54.875824 25100 log.cc:824] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:54.877458 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: No bootstrap required, opened a new log
I20260924 10:48:54.880237 25100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.880425 25100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455f9158727844e190aba64f18a910fd, State: Initialized, Role: FOLLOWER
I20260924 10:48:54.880722 25100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.881175 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.881306 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.881881 25100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:54.882011 25074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:48:54.882100 25074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:54.883410 25074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.884294 25074 leader_election.cc:288] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [CANDIDATE]: Term 1 election: Requested vote from peers 31473d133490415fa07f599169486a74 (127.2.182.61:41421), c4a8514b274f4263a6ff245798d5d546 (127.2.182.60:43453)
I20260924 10:48:54.885353 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "455f9158727844e190aba64f18a910fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31473d133490415fa07f599169486a74"
I20260924 10:48:54.885560 25149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:54.886860 25149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455f9158727844e190aba64f18a910fd in term 1.
W20260924 10:48:54.887048 25295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:54.887115 25295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:54.887306 25048 leader_election.cc:302] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31473d133490415fa07f599169486a74, 455f9158727844e190aba64f18a910fd; no voters:
I20260924 10:48:54.887279 25224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "455f9158727844e190aba64f18a910fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4a8514b274f4263a6ff245798d5d546"
I20260924 10:48:54.887502 25224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:54.887560 25284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:54.887735 25284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [term 1 LEADER]: Becoming Leader. State: Replica: 455f9158727844e190aba64f18a910fd, State: Running, Role: LEADER
I20260924 10:48:54.888026 25284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } }
I20260924 10:48:54.888724 25224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455f9158727844e190aba64f18a910fd in term 1.
I20260924 10:48:54.889750 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455f9158727844e190aba64f18a910fd. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.889853 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:54.892339 25303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 10:48:54.893793 25053 tablet.cc:2369] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:54.894227 25303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:54.894501 25032 master_runner.cc:263] No leader master found from master 455f9158727844e190aba64f18a910fd
I20260924 10:48:54.894562 25032 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:48:54.900046 25224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [term 1 FOLLOWER]: Refusing update from remote peer 455f9158727844e190aba64f18a910fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:54.900398 25149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [term 1 FOLLOWER]: Refusing update from remote peer 455f9158727844e190aba64f18a910fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:54.900635 25301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:54.901031 25301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:54.904146 25256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455f9158727844e190aba64f18a910fd. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.904265 25256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.904397 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455f9158727844e190aba64f18a910fd. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.904531 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.906720 25301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.906857 25301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:54.907328 25307 mvcc.cc:205] Tried to move back new op lower bound from 7332851445343277056 to 7332851445304557568. Current Snapshot: MvccSnapshot[applied={T|T < 7332851445343277056}]
I20260924 10:48:54.907555 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.907680 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:54.908036 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.890173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.2.182.62:45603)
0924 10:48:54.890274 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.890278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.890823 (+ 545us) server_negotiation.cc:407] Connection header received
0924 10:48:54.890999 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.891001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.891043 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.891090 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.892074 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.892775 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.895657 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.895958 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.896033 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.897874 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.897877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.897880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.897903 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.899558 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.905101 (+ 5543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.905105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.905107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.905192 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.905513 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.905518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.905520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.905754 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.905866 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.907776 (+ 1910us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.907890 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:48:54.908622 25256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.908715 25256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.909775 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "455f9158727844e190aba64f18a910fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455f9158727844e190aba64f18a910fd" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 45603 } } peers { permanent_uuid: "31473d133490415fa07f599169486a74" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 41421 } } peers { permanent_uuid: "c4a8514b274f4263a6ff245798d5d546" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 43453 } } }
I20260924 10:48:54.909883 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:54.911813 25309 mvcc.cc:205] Tried to move back new op lower bound from 7332851445343277056 to 7332851445304557568. Current Snapshot: MvccSnapshot[applied={T|T < 7332851445343277056}]
I20260924 10:48:54.912462 25308 mvcc.cc:205] Tried to move back new op lower bound from 7332851445343277056 to 7332851445304557568. Current Snapshot: MvccSnapshot[applied={T|T < 7332851445343277056}]
I20260924 10:48:54.912962 25303 catalog_manager.cc:1347] Generated new cluster ID: 5ecd481982104bad948192aff85490e7
I20260924 10:48:54.913019 25303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:54.917991 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.893215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.2.182.61:41421)
0924 10:48:54.893318 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.893323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.901127 (+ 7804us) server_negotiation.cc:407] Connection header received
0924 10:48:54.901299 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.901302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.901350 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.901398 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.902190 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.902957 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.904082 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.904416 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.904489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.913998 (+ 9509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.914003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.914006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.914032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.916112 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.916673 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.916676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.916677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.916751 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.917019 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.917022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.917023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.917195 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.917307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.917762 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.917861 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:48:54.918200 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:54.894409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.2.182.60:43453)
0924 10:48:54.894525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.894529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.909144 (+ 14615us) server_negotiation.cc:407] Connection header received
0924 10:48:54.909299 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.909302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.909345 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.909415 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.910129 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.910828 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.911930 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:54.912248 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:54.912312 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:54.914506 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:54.914509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:54.914512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:54.914535 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:54.916115 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.916705 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.916709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.916711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.916790 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:54.917094 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:54.917098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:54.917100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:54.917288 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:54.917393 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:54.917960 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 10:48:54.918065 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:48:54.924587 25303 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:54.925719 25303 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:54.947759 25303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 455f9158727844e190aba64f18a910fd: Generated new TSK 0
I20260924 10:48:54.948742 25303 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:48:54.977530 25104 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.975661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.2.182.62:45603)
0924 10:48:54.975801 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.975805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.976451 (+ 646us) server_negotiation.cc:407] Connection header received
0924 10:48:54.976644 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.976648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.976698 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.976763 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.977398 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53530: BlockingRecv error: recv got EOF from 127.0.0.1:53530 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 10:48:54.979398 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.978192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.2.182.61:41421)
0924 10:48:54.978300 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.978305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.978533 (+ 228us) server_negotiation.cc:407] Connection header received
0924 10:48:54.978698 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.978701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.978757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.978807 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.979281 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39224: BlockingRecv error: recv got EOF from 127.0.0.1:39224 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 10:48:54.980783 25253 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.979938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.2.182.60:43453)
0924 10:48:54.980068 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.980073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.980088 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:48:54.980180 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.980183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.980234 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.980319 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.980670 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36498: BlockingRecv error: recv got EOF from 127.0.0.1:36498 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 10:48:54.984483 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.983008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.2.182.61:41421)
0924 10:48:54.983100 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.983104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.983583 (+ 479us) server_negotiation.cc:407] Connection header received
0924 10:48:54.983800 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.983803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.983875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.983927 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.984390 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39232: BlockingRecv error: recv got EOF from 127.0.0.1:39232 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 10:48:54.991338 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.988087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.2.182.61:41421)
0924 10:48:54.988193 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.988197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.988678 (+ 481us) server_negotiation.cc:407] Connection header received
0924 10:48:54.990189 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.990194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.990269 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.990330 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.991213 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39246: BlockingRecv error: recv got EOF from 127.0.0.1:39246 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:48:54.997350 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:54.995734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.2.182.61:41421)
0924 10:48:54.995842 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:54.995847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:54.996349 (+ 502us) server_negotiation.cc:407] Connection header received
0924 10:48:54.996550 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:54.996553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:54.996606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:54.996663 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:54.997237 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39260: BlockingRecv error: recv got EOF from 127.0.0.1:39260 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:48:55.005427 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.003751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.2.182.61:41421)
0924 10:48:55.003899 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.003904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.004407 (+ 503us) server_negotiation.cc:407] Connection header received
0924 10:48:55.004612 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.004616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.004669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.004738 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.005311 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39276: BlockingRecv error: recv got EOF from 127.0.0.1:39276 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 10:48:55.017405 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.015859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.2.182.61:41421)
0924 10:48:55.015963 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.015967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.016484 (+ 517us) server_negotiation.cc:407] Connection header received
0924 10:48:55.016665 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.016668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.016716 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.016767 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.017300 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39292: BlockingRecv error: recv got EOF from 127.0.0.1:39292 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 10:48:55.037392 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.035749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.2.182.61:41421)
0924 10:48:55.035860 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.035866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.036391 (+ 525us) server_negotiation.cc:407] Connection header received
0924 10:48:55.036604 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.036608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.036664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.036718 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.037291 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39298: BlockingRecv error: recv got EOF from 127.0.0.1:39298 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:48:55.074172 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.072155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.2.182.61:41421)
0924 10:48:55.072303 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.072308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.073031 (+ 723us) server_negotiation.cc:407] Connection header received
0924 10:48:55.073310 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.073314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.073369 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.073426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.074049 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39308: BlockingRecv error: recv got EOF from 127.0.0.1:39308 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:48:55.142112 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.141003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.2.182.61:41421)
0924 10:48:55.141166 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.141170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.141261 (+ 91us) server_negotiation.cc:407] Connection header received
0924 10:48:55.141388 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.141391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.141438 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.141516 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.141966 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39316: BlockingRecv error: recv got EOF from 127.0.0.1:39316 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 10:48:55.273097 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.272038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.2.182.61:41421)
0924 10:48:55.272169 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.272173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.272334 (+ 161us) server_negotiation.cc:407] Connection header received
0924 10:48:55.272466 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.272469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.272515 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.272604 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.272997 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39332: BlockingRecv error: recv got EOF from 127.0.0.1:39332 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 10:48:55.532384 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:55.531293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.2.182.61:41421)
0924 10:48:55.531449 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:55.531456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:55.531524 (+ 68us) server_negotiation.cc:407] Connection header received
0924 10:48:55.531690 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:55.531693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:55.531744 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:55.531828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:55.532260 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39336: BlockingRecv error: recv got EOF from 127.0.0.1:39336 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:48:55.861330 25268 catalog_manager.cc:1259] Loaded cluster ID: 5ecd481982104bad948192aff85490e7
I20260924 10:48:55.861413 25268 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: loading cluster ID for follower catalog manager: success
I20260924 10:48:55.863160 25268 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: acquiring CA information for follower catalog manager: success
I20260924 10:48:55.864451 25268 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4a8514b274f4263a6ff245798d5d546: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:48:55.870096 25281 catalog_manager.cc:1259] Loaded cluster ID: 5ecd481982104bad948192aff85490e7
I20260924 10:48:55.870149 25281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: loading cluster ID for follower catalog manager: success
I20260924 10:48:55.871567 25281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: acquiring CA information for follower catalog manager: success
I20260924 10:48:55.872536 25281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31473d133490415fa07f599169486a74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:48:56.050451 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:56.046354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.2.182.61:41421)
0924 10:48:56.046601 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:56.046606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:56.046713 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:48:56.046934 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:56.046937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:56.047075 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:56.047195 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:56.047463 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.047994 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.048882 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.049133 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.049245 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:56.049562 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:56.050034 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:56.050167 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:56.050211 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 10:48:56.050284 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260924 10:48:56.056985 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:56.053321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.2.182.60:43453)
0924 10:48:56.053565 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:56.053569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:56.053582 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:48:56.053735 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:56.053741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:56.053923 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:56.054021 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:56.054192 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.054728 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.055650 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.055924 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.056008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:56.056233 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:56.056603 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:56.056687 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:56.056740 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:48:56.056783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260924 10:48:56.063503 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:56.060269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.2.182.60:43453)
0924 10:48:56.060370 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:56.060374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:56.060536 (+ 162us) server_negotiation.cc:407] Connection header received
0924 10:48:56.060623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:56.060625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:56.060671 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:56.060734 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:56.061067 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.061506 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.062331 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.062538 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.062641 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:56.063053 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:56.063125 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:56.063170 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:56.063337 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 10:48:56.063382 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:48:56.063896 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:56.059775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.2.182.61:41421)
0924 10:48:56.059890 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:56.059896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:56.060141 (+ 245us) server_negotiation.cc:407] Connection header received
0924 10:48:56.060331 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:56.060335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:56.060383 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:56.060430 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:56.060758 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.061463 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.062607 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.062947 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.063041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:56.063354 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:56.063424 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:56.063491 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:56.063707 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 10:48:56.063760 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:48:56.064412 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:56.059708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.2.182.62:45603)
0924 10:48:56.059955 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:56.059959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:56.060214 (+ 255us) server_negotiation.cc:407] Connection header received
0924 10:48:56.060302 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:56.060305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:56.060462 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:56.060581 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:56.060969 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.061646 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.062656 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:56.062911 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:56.063030 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:56.063332 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:56.063694 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:56.063772 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:56.064185 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 10:48:56.064239 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260924 10:48:56.072046 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25032
I20260924 10:48:56.078190 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25107
I20260924 10:48:56.083811 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25182
2026-09-24T10:48:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2990 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:48:56 dist-test-slave-6lwt krb5kdc[25396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:48:56 dist-test-slave-6lwt krb5kdc[25396](info): set up 2 sockets
Sep 24 10:48:56 dist-test-slave-6lwt krb5kdc[25396](info): commencing operation
krb5kdc: starting...
W20260924 10:48:57.085572 2776 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/test-user.keytab.
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
Entry for principal kudu/127.2.182.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab.
I20260924 10:48:57.200528 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:34761
--webserver_interface=127.2.182.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:48:57.306071 25409 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:57.306357 25409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:57.306406 25409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:57.306474 25409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:57.310446 25409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:57.310516 25409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:57.310555 25409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:57.310592 25409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:57.310629 25409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:57.315584 25409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.62.keytab
--principal=kudu/127.2.182.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:57.316510 25409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:57.317464 25409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:57.325043 25415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:57.325047 25414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:57.325141 25409 server_base.cc:1034] running on GCE node
W20260924 10:48:57.325330 25417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:57.325793 25409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:57.326786 25409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:57.327930 25409 hybrid_clock.cc:648] HybridClock initialized: now 1790246937327906 us; error 36 us; skew 500 ppm
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:57.330348 25409 init.cc:377] Logged in from keytab as kudu/127.2.182.62@KRBTEST.COM (short username kudu)
I20260924 10:48:57.331048 25409 webserver.cc:460] Webserver started at http://127.2.182.62:36387/ using document root <none> and password file <none>
I20260924 10:48:57.331507 25409 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:57.331555 25409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:57.331709 25409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:57.333613 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data/instance:
uuid: "4ce14eb86dda438fb0343810593f2c6b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "bca20075c4d20246ca778994af353fa5"
server_key_iv: "4916d48c3a3e8bf6e4e43d911c7adcf1"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.334115 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance:
uuid: "4ce14eb86dda438fb0343810593f2c6b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "bca20075c4d20246ca778994af353fa5"
server_key_iv: "4916d48c3a3e8bf6e4e43d911c7adcf1"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.337342 25409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:48:57.339313 25424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:57.340384 25409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:48:57.340526 25409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal
uuid: "4ce14eb86dda438fb0343810593f2c6b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "bca20075c4d20246ca778994af353fa5"
server_key_iv: "4916d48c3a3e8bf6e4e43d911c7adcf1"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.340622 25409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:57.371863 25409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:57.374750 25409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:57.374935 25409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:57.381326 25409 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:34761
I20260924 10:48:57.381618 25476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.62:34761 every 8 connection(s)
I20260924 10:48:57.382273 25409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/data/info.pb
I20260924 10:48:57.384824 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:57.385771 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25409
I20260924 10:48:57.385912 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-0/wal/instance
I20260924 10:48:57.386179 2776 external_mini_cluster.cc:1428] Setting key 96882a5feef8286ce05da3be851f158f
I20260924 10:48:57.388756 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.182.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.61@KRBTEST.COM" created.
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:57.403265 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.390810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34761 (local address 127.0.0.1:46042)
0924 10:48:57.391231 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.391289 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.392117 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.392956 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.392960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.393261 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.393553 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.393565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.394886 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.394888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.395644 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.395653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.396249 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.396252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.396355 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.396885 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.396935 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.398440 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.400621 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.400628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.400634 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.400845 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.401236 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.401238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.401239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.401341 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.402067 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.402072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.402174 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.402291 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.402495 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
I20260924 10:48:57.403270 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.390810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.2.182.62:34761)
0924 10:48:57.391359 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.391373 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.391433 (+ 60us) server_negotiation.cc:407] Connection header received
0924 10:48:57.392428 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.392436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.392705 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.392961 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.393696 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.394785 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.395735 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.396126 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.396244 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.398573 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.398578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.398587 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.398607 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.400520 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.400986 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.400995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.400998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.401141 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.401451 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.401457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.401461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.401962 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.402107 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.402419 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.402573 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":423,"thread_start_us":157,"threads_started":1}
I20260924 10:48:57.405704 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:57.406363 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.406155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36414)
0924 10:48:57.406251 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.406315 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:48:57.406533 25426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
I20260924 10:48:57.407256 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.407072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36430)
0924 10:48:57.407169 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.407215 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:48:57.407667 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:43701: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
Entry for principal kudu/127.2.182.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab.
I20260924 10:48:57.412698 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:43701
--webserver_interface=127.2.182.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:57.456943 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } attempt: 1
I20260924 10:48:57.457748 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.457484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36438)
0924 10:48:57.457624 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.457680 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:48:57.458457 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.458259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36440)
0924 10:48:57.458355 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.458404 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:48:57.458608 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:43701: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
W20260924 10:48:57.518692 25484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:57.518945 25484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:57.518966 25484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:57.519003 25484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:57.522826 25484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:57.522876 25484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:57.522894 25484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:57.522910 25484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:57.522979 25484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:57.526762 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } attempt: 2
I20260924 10:48:57.527570 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.527293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36444)
0924 10:48:57.527439 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.527499 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:48:57.528232 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.528057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36454)
0924 10:48:57.528146 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.528185 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 10:48:57.528497 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.61:43701: Network error: Client connection negotiation failed: client connection to 127.2.182.61:43701: connect: Connection refused (error 111)
I20260924 10:48:57.528668 25484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.61.keytab
--principal=kudu/127.2.182.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.61:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:57.529666 25484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:57.530607 25484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:48:57.538621 25489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:57.538543 25490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:57.538537 25492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:57.538847 25484 server_base.cc:1034] running on GCE node
I20260924 10:48:57.539428 25484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:57.540467 25484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:57.541643 25484 hybrid_clock.cc:648] HybridClock initialized: now 1790246937541624 us; error 27 us; skew 500 ppm
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:57.544406 25484 init.cc:377] Logged in from keytab as kudu/127.2.182.61@KRBTEST.COM (short username kudu)
I20260924 10:48:57.545202 25484 webserver.cc:460] Webserver started at http://127.2.182.61:42327/ using document root <none> and password file <none>
I20260924 10:48:57.545679 25484 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:57.545732 25484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:57.545881 25484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:57.547801 25484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data/instance:
uuid: "a53ba7d8de2d4a76a3721793dda0b7bc"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "f5db72401ae5369bf7da82c20ffcca06"
server_key_iv: "1d71b9181d07f783b6d49bf86e20d09e"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.548292 25484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal/instance:
uuid: "a53ba7d8de2d4a76a3721793dda0b7bc"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "f5db72401ae5369bf7da82c20ffcca06"
server_key_iv: "1d71b9181d07f783b6d49bf86e20d09e"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.551677 25484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:48:57.553867 25499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:57.554996 25484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:57.555127 25484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal
uuid: "a53ba7d8de2d4a76a3721793dda0b7bc"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "f5db72401ae5369bf7da82c20ffcca06"
server_key_iv: "1d71b9181d07f783b6d49bf86e20d09e"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.555227 25484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:57.574278 25484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:57.577032 25484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:57.577229 25484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:57.583696 25484 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.61:43701
I20260924 10:48:57.583969 25551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.61:43701 every 8 connection(s)
I20260924 10:48:57.584671 25484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/data/info.pb
I20260924 10:48:57.587334 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:57.588131 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25484
I20260924 10:48:57.588268 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-1/wal/instance
I20260924 10:48:57.588553 2776 external_mini_cluster.cc:1428] Setting key dff1586a30cf1cb1ddf0a8e825d6e02c
I20260924 10:48:57.591540 25552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.182.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.60@KRBTEST.COM" created.
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:57.605408 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.593430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.2.182.62:34761)
0924 10:48:57.593559 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.593562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.594588 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 10:48:57.595461 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.595464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.595525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.595616 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.596753 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.597338 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.598411 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.598686 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.598793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.601533 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.601536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.601538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.601561 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.603062 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.603719 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.603722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.603723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.603792 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.604155 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.604157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.604158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.604362 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.604439 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.605131 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.605230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:57.606581 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.593960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34761 (local address 127.0.0.1:46058)
0924 10:48:57.594435 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.594487 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.595259 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.595770 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.595777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.596118 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.596612 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.596625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.597472 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.597480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.598282 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.598290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.598824 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.598828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.599005 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.599597 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.599613 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.601359 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.603181 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.603190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.603209 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.603615 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.603877 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.603880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.603882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.604064 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.604472 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.604480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.604820 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.605079 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.605440 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260924 10:48:57.607890 25552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
Entry for principal kudu/127.2.182.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab.
I20260924 10:48:57.615830 2776 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
/tmp/dist-test-taskwg6lLW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:46805
--webserver_interface=127.2.182.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:48:57.617093 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.608384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36466)
0924 10:48:57.608518 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.608549 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.608640 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.609240 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.609242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.609428 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.609532 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.609537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.610436 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.610439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.611027 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.611036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.611790 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.611793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.611954 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.612456 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.612470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.613755 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.615615 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.615620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.615623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.615811 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.616137 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.616140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.616142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.616207 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.616618 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.616621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.616694 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.616790 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.616934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":2}
I20260924 10:48:57.617153 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.608533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.2.182.61:43701)
0924 10:48:57.608844 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.608859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.608878 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:48:57.608945 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.608951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.609105 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.609217 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.609627 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.610338 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.611180 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.611668 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.611787 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.613881 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.613890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.613903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.613928 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.615522 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.615913 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.615918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.615920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.616044 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.616285 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.616290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.616292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.616522 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.616644 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.616881 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.617022 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":222,"thread_start_us":145,"threads_started":1}
I20260924 10:48:57.618801 25552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:57.619443 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.619216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37828)
0924 10:48:57.619314 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.619388 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:48:57.619623 25503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
I20260924 10:48:57.619664 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } attempt: 3
I20260924 10:48:57.620410 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.620164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37844)
0924 10:48:57.620288 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.620331 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 10:48:57.620543 25552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
I20260924 10:48:57.628328 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.620103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36482)
0924 10:48:57.620250 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.620264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.620343 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.620710 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.620712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.620884 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.621023 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.621029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.622108 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.622110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.622578 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.622584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.623247 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.623250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.623325 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.623762 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.623780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.625235 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.627141 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.627143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.627144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.627312 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.627592 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.627594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.627595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.627672 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.627998 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.628000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.628068 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.628137 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.628228 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:48:57.628432 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.620359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.2.182.61:43701)
0924 10:48:57.620492 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.620497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.620512 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:48:57.620551 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.620555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.620605 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.620697 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.621131 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.622009 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.622677 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.623139 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.623266 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.625360 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.625366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.625372 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.625414 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.627033 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.627427 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.627431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.627433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.627510 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.627741 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.627744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.627745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.627919 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.628005 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.628198 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.628295 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:48:57.629325 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:57.630290 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.630047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37848)
0924 10:48:57.630173 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.630223 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:48:57.630946 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.630721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37852)
0924 10:48:57.630844 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.630890 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 10:48:57.631060 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
I20260924 10:48:57.662216 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 1
I20260924 10:48:57.663015 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.662680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37866)
0924 10:48:57.662853 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.662919 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:48:57.663678 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.663474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37882)
0924 10:48:57.663572 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.663621 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:48:57.663808 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
I20260924 10:48:57.669737 25552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 1
I20260924 10:48:57.670395 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.670162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37892)
0924 10:48:57.670279 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.670327 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 10:48:57.671065 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.670895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37894)
0924 10:48:57.670981 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.671020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 10:48:57.671238 25552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
W20260924 10:48:57.724745 25558 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:48:57.725054 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:48:57.725104 25558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:48:57.725160 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:48:57.728942 25558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:48:57.728994 25558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:48:57.729012 25558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:48:57.729051 25558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:48:57.729081 25558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:48:57.733906 25558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:34761,127.2.182.61:43701,127.2.182.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/krb5kdc/kudu_127.2.182.60.keytab
--principal=kudu/127.2.182.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.60:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:48:57.734859 25558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:48:57.735831 25558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:48:57.738962 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 2
I20260924 10:48:57.739387 25552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 2
I20260924 10:48:57.739892 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.739526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37902)
0924 10:48:57.739731 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.739799 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 10:48:57.740156 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.739816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37910)
0924 10:48:57.740004 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.740079 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 10:48:57.740655 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.740414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37924)
0924 10:48:57.740541 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.740585 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:48:57.740778 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.740569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37938)
0924 10:48:57.740674 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.740720 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:48:57.740778 25477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
W20260924 10:48:57.740885 25552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.60:46805: Network error: Client connection negotiation failed: client connection to 127.2.182.60:46805: connect: Connection refused (error 111)
W20260924 10:48:57.744159 25564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:57.744158 25567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:48:57.744186 25565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:48:57.744333 25558 server_base.cc:1034] running on GCE node
I20260924 10:48:57.745007 25558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:48:57.746138 25558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:48:57.747336 25558 hybrid_clock.cc:648] HybridClock initialized: now 1790246937747310 us; error 35 us; skew 500 ppm
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:48:57.749998 25558 init.cc:377] Logged in from keytab as kudu/127.2.182.60@KRBTEST.COM (short username kudu)
I20260924 10:48:57.750816 25558 webserver.cc:460] Webserver started at http://127.2.182.60:32813/ using document root <none> and password file <none>
I20260924 10:48:57.751380 25558 fs_manager.cc:359] Metadata directory not provided
I20260924 10:48:57.751492 25558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:48:57.751695 25558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:48:57.753649 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data/instance:
uuid: "3d2ce8f151d64cf48ec8f15c8742973b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "92d97d85756348f98dd3bceb37765bb0"
server_key_iv: "beba89229792b56eff9ec2d75c39230a"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.754181 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal/instance:
uuid: "3d2ce8f151d64cf48ec8f15c8742973b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "92d97d85756348f98dd3bceb37765bb0"
server_key_iv: "beba89229792b56eff9ec2d75c39230a"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.757583 25558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:48:57.759846 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:48:57.761102 25558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:48:57.761302 25558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data,/tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal
uuid: "3d2ce8f151d64cf48ec8f15c8742973b"
format_stamp: "Formatted at 2026-09-24 10:48:57 on dist-test-slave-6lwt"
server_key: "92d97d85756348f98dd3bceb37765bb0"
server_key_iv: "beba89229792b56eff9ec2d75c39230a"
server_key_version: "encryptionkey@0"
I20260924 10:48:57.761391 25558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:48:57.774734 25558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:48:57.777676 25558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:48:57.777818 25558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:48:57.784412 25558 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.60:46805
I20260924 10:48:57.784673 25626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.60:46805 every 8 connection(s)
I20260924 10:48:57.785346 25558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/data/info.pb
I20260924 10:48:57.787951 25627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:48:57.791374 25627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:57.791797 2776 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu as pid 25558
I20260924 10:48:57.791939 2776 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwg6lLW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246662595599-2776-0/minicluster-data/master-2/wal/instance
I20260924 10:48:57.792239 2776 external_mini_cluster.cc:1428] Setting key b8f357af5f4962d3a7f996c11d5c719a
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:48:57.803687 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.793331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.2.182.62:34761)
0924 10:48:57.793450 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.793454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.794013 (+ 559us) server_negotiation.cc:407] Connection header received
0924 10:48:57.794789 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.794793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.794852 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.794967 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.795828 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.796265 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.797340 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.797560 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.797633 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.800033 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.800039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.800044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.800073 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.801553 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.802096 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.802099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.802100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.802168 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.802519 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.802523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.802525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.802740 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.802836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.803389 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.803520 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:48:57.804527 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.793421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.62:34761 (local address 127.0.0.1:46062)
0924 10:48:57.793880 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.793922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.794598 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.795030 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.795035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.795421 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.795689 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.795701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.796361 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.796365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.797238 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.797247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.797685 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.797688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.797837 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.798584 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.798597 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.799845 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.801669 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.801675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.801683 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.802002 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.802250 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.802253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.802256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.802414 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.802847 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.802853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.803136 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.803329 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.803604 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":44,"spinlock_wait_cycles":12160,"thread_start_us":136,"threads_started":1}
I20260924 10:48:57.804526 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.793421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.2.182.60:46805)
0924 10:48:57.793809 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.793825 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.793862 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:48:57.794583 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.794600 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.794789 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.794958 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.795650 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.796533 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.797376 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.797745 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.797837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.800415 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.800422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.800433 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.800459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.802314 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.802770 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.802779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.802780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.802873 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.803136 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.803139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.803140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.803398 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.803562 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.803997 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.804120 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260924 10:48:57.805712 25627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.62@KRBTEST.COM
I20260924 10:48:57.814200 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.806113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.61:43701 (local address 127.0.0.1:36486)
0924 10:48:57.806195 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.806208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.806274 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.806747 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.806750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.806909 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.807008 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.807014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.807884 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.807887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.808559 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.808567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.809145 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.809149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.809256 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.809755 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.809764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.810849 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.812685 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.812689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.812691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.812979 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.813234 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.813239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.813242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.813335 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.813733 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.813735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.813804 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.813921 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.814062 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 10:48:57.814303 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.806228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.2.182.61:43701)
0924 10:48:57.806352 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.806356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.806375 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:48:57.806565 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.806568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.806645 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.806736 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.807118 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.807796 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.808671 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.809038 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.809147 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.811001 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.811006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.811009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.811034 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.812585 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.813067 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.813070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.813071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.813143 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.813420 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.813422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.813423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.813629 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.813728 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.813976 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.814100 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:48:57.814849 25627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:48:57.815186 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.806737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.2.182.62:34761)
0924 10:48:57.806897 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.806901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.806934 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:48:57.806962 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.806965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.807002 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.807080 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.807662 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.808240 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.809135 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.809482 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.809559 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.811848 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.811851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.811858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.811878 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.813463 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.813906 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.813912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.813916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.813995 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.814333 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.814338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.814339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.814544 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.814787 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.814936 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.815038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.60@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.61@KRBTEST.COM
I20260924 10:48:57.822371 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.815311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37960)
0924 10:48:57.815425 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.815440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.815525 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.815760 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.815762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.815922 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.816009 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.816017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.816831 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.816837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.817346 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.817354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.817767 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.817770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.817859 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.818271 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.818281 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.819239 (+ 958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.820942 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.820945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.820951 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.821123 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.821380 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.821384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.821386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.821475 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.821958 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.821961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.822035 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.822127 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.822233 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:48:57.822957 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.815338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.2.182.60:46805)
0924 10:48:57.815434 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.815440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.815485 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:48:57.815605 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.815609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.815648 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.815736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.816127 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.816756 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.817458 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.817678 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.817754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.819388 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.819391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.819394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.819415 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.820799 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.821228 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.821230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.821231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.821298 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.821564 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.821567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.821568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.821790 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.821922 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.822691 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.822787 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 10:48:57.825253 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.816593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.2.182.61:43701)
0924 10:48:57.816687 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.816691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.816703 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:48:57.816759 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.816762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.816794 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.816856 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.817452 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.817981 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.818750 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.818948 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.819037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.821611 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.821616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.821619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.821643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.823677 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.824141 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.824144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.824146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.824223 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.824481 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.824485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.824487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.824650 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.824860 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.825003 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.825103 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 10:48:57.832113 25552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 3
I20260924 10:48:57.833119 25627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: Bootstrap starting.
I20260924 10:48:57.833411 2776 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:48:57.834509 25451 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4ce14eb86dda438fb0343810593f2c6b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46076
I20260924 10:48:57.835851 25627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:57.836750 25627 log.cc:824] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.61@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:48:57.838912 25627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: No bootstrap required, opened a new log
I20260924 10:48:57.839912 25477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } attempt: 3
I20260924 10:48:57.841019 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.832434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37972)
0924 10:48:57.832526 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.832542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.832612 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.833079 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.833082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.833255 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.833344 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.833350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.834411 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.834414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.835120 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.835131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.835678 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.835684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.835788 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.836221 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.836234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.837665 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.839383 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.839386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.839388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.839594 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.839861 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.839865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.839867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.839983 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.840497 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.840500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.840592 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.840699 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.840814 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 10:48:57.841646 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.832726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.2.182.60:46805)
0924 10:48:57.832842 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.832846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.832858 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:48:57.832887 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.832890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.832953 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.833527 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.833604 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.834254 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.835258 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.835552 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.836073 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.837772 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.837775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.837778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.837799 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.839285 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.839693 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.839697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.839698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.839775 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.840135 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.840141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.840145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.840376 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.841363 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.841426 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.841517 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:48:57.842948 25627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.843214 25627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2ce8f151d64cf48ec8f15c8742973b, State: Initialized, Role: FOLLOWER
I20260924 10:48:57.843571 25627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.844008 25634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.844135 25634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.844457 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: Bootstrap starting.
I20260924 10:48:57.844967 25627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 10:48:57 dist-test-slave-6lwt krb5kdc[25396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.62@KRBTEST.COM for kudu/127.2.182.60@KRBTEST.COM
I20260924 10:48:57.846889 25552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:57.847718 25552 log.cc:824] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: Log is configured to *not* fsync() on all Append() calls
W20260924 10:48:57.849962 25645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:57.850075 25645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:57.850225 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.840260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.60:46805 (local address 127.0.0.1:37984)
0924 10:48:57.840447 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:48:57.840463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:48:57.840531 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:48:57.841014 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:48:57.841017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:48:57.841181 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:48:57.841276 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.841282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.842202 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.842205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.843003 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:48:57.843014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.843601 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.843605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:48:57.843686 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.844323 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:48:57.844336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:48:57.845745 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:48:57.848604 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.848606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.848610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.848928 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.849259 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:48:57.849262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:48:57.849264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:48:57.849343 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:48:57.849761 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:48:57.849764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:48:57.849851 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:48:57.849944 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:48:57.850066 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 10:48:57.850664 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.840311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.2.182.60:46805)
0924 10:48:57.840786 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.840796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.840807 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:48:57.840834 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.840836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.840871 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.840988 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.841388 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.842099 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.843127 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.843493 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.843600 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.845882 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.845885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.845887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.845909 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.848494 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.849078 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.849081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.849083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.849162 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.849435 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.849438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.849439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.849613 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.849698 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.850402 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.850512 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":77,"threads_started":1}
I20260924 10:48:57.848801 25558 master_runner.cc:186] Error getting master registration for 127.2.182.62:34761: OK, instance_id { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" instance_seqno: 1790246937379598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:48:57.852083 25558 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:48:57.852414 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: No bootstrap required, opened a new log
I20260924 10:48:57.853446 25477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: Bootstrap starting.
I20260924 10:48:57.855808 25477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: Neither blocks nor log segments found. Creating new log.
I20260924 10:48:57.856372 25552 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.856585 25552 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53ba7d8de2d4a76a3721793dda0b7bc, State: Initialized, Role: FOLLOWER
I20260924 10:48:57.856570 25477 log.cc:824] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:48:57.856930 25552 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.858246 25552 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:57.858440 25477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: No bootstrap required, opened a new log
I20260924 10:48:57.861305 25477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.861511 25477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce14eb86dda438fb0343810593f2c6b, State: Initialized, Role: FOLLOWER
I20260924 10:48:57.861822 25477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.863036 25477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:48:57.863090 25451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:48:57.863176 25451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:57.864516 25451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
I20260924 10:48:57.865223 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.865342 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.865532 25451 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [CANDIDATE]: Term 1 election: Requested vote from peers a53ba7d8de2d4a76a3721793dda0b7bc (127.2.182.61:43701), 3d2ce8f151d64cf48ec8f15c8742973b (127.2.182.60:46805)
I20260924 10:48:57.861852 25484 master_runner.cc:186] Error getting master registration for 127.2.182.62:34761: OK, instance_id { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" instance_seqno: 1790246937379598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:48:57.865963 25484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:48:57.866214 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ce14eb86dda438fb0343810593f2c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d2ce8f151d64cf48ec8f15c8742973b"
I20260924 10:48:57.866256 25526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ce14eb86dda438fb0343810593f2c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc"
I20260924 10:48:57.866453 25526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:57.866453 25601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:48:57.867702 25526 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce14eb86dda438fb0343810593f2c6b in term 1.
I20260924 10:48:57.867702 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce14eb86dda438fb0343810593f2c6b in term 1.
I20260924 10:48:57.868109 25428 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2ce8f151d64cf48ec8f15c8742973b, 4ce14eb86dda438fb0343810593f2c6b; no voters:
I20260924 10:48:57.868561 25674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.868743 25674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.868868 25674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:48:57.869026 25674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [term 1 LEADER]: Becoming Leader. State: Replica: 4ce14eb86dda438fb0343810593f2c6b, State: Running, Role: LEADER
I20260924 10:48:57.869347 25674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } }
W20260924 10:48:57.870486 25660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:57.870563 25660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:57.875159 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce14eb86dda438fb0343810593f2c6b. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.875270 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: This master's current role is: LEADER
W20260924 10:48:57.875483 25673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:48:57.875547 25673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:48:57.875707 25682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 10:48:57.876356 25430 tablet.cc:2369] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:48:57.877187 25682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:48:57.882273 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.871348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.2.182.61:43701)
0924 10:48:57.871463 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.871468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.871972 (+ 504us) server_negotiation.cc:407] Connection header received
0924 10:48:57.872391 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.872394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.872453 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.872502 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.873511 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.874273 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.876143 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.876482 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.876553 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.878451 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.878455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.878458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.878487 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.880212 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.880733 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.880737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.880739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.880822 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.881201 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.881204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.881206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.881447 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.881571 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.881946 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.882065 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:48:57.891412 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.881239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.2.182.62:34761)
0924 10:48:57.881350 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.881354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.882089 (+ 735us) server_negotiation.cc:407] Connection header received
0924 10:48:57.882303 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.882306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.882354 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.882408 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.883531 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.884308 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.885509 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.885840 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.885915 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.887689 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.887693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.887696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.887721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.889522 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.890031 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.890035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.890037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.890123 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.890439 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.890442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.890444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.890647 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.890757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.891109 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.891248 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:48:57.897984 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:57.881692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.2.182.60:46805)
0924 10:48:57.881816 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:57.881822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:57.882175 (+ 353us) server_negotiation.cc:407] Connection header received
0924 10:48:57.882362 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:57.882365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:57.882418 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:57.882477 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:57.885033 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.885810 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.892194 (+ 6384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:57.892587 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:57.892667 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:57.894461 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:48:57.894466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:48:57.894469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:48:57.894497 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:48:57.896178 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.896648 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.896651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.896653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.896730 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:48:57.897045 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:48:57.897048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:48:57.897050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:48:57.897259 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:48:57.897381 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:57.897703 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:48:57.897827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:48:57.900360 25601 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [term 1 FOLLOWER]: Refusing update from remote peer 4ce14eb86dda438fb0343810593f2c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:57.901289 25661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:57.901682 25526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [term 1 FOLLOWER]: Refusing update from remote peer 4ce14eb86dda438fb0343810593f2c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:48:57.905531 25661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:48:57.909338 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce14eb86dda438fb0343810593f2c6b. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.909457 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.913668 25634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce14eb86dda438fb0343810593f2c6b. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.913791 25634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.917898 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.918011 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:57.918109 25683 mvcc.cc:205] Tried to move back new op lower bound from 7332851457623674880 to 7332851457506500608. Current Snapshot: MvccSnapshot[applied={T|T < 7332851457623674880}]
I20260924 10:48:57.919255 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.919353 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.919674 25674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.919788 25674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b [sys.catalog]: This master's current role is: LEADER
I20260924 10:48:57.920629 25634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ce14eb86dda438fb0343810593f2c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce14eb86dda438fb0343810593f2c6b" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 34761 } } peers { permanent_uuid: "a53ba7d8de2d4a76a3721793dda0b7bc" member_type: VOTER last_known_addr { host: "127.2.182.61" port: 43701 } } peers { permanent_uuid: "3d2ce8f151d64cf48ec8f15c8742973b" member_type: VOTER last_known_addr { host: "127.2.182.60" port: 46805 } } }
I20260924 10:48:57.920747 25634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:48:57.920977 25687 mvcc.cc:205] Tried to move back new op lower bound from 7332851457623674880 to 7332851457506500608. Current Snapshot: MvccSnapshot[applied={T|T < 7332851457623674880}]
I20260924 10:48:57.921885 25686 mvcc.cc:205] Tried to move back new op lower bound from 7332851457623674880 to 7332851457506500608. Current Snapshot: MvccSnapshot[applied={T|T < 7332851457623674880}]
I20260924 10:48:57.923878 25682 catalog_manager.cc:1347] Generated new cluster ID: 92cf71df70294679b7c946ccbd4e14ce
I20260924 10:48:57.923940 25682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:48:57.951557 25682 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:48:57.952733 25682 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:48:57.971280 25682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce14eb86dda438fb0343810593f2c6b: Generated new TSK 0
I20260924 10:48:57.974216 25682 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:48:58.007308 25633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.004330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.2.182.60:46805)
0924 10:48:58.004462 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.004468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.006272 (+ 1804us) server_negotiation.cc:407] Connection header received
0924 10:48:58.006495 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.006499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.006545 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.006594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.007182 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37992: BlockingRecv error: recv got EOF from 127.0.0.1:37992 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 10:48:58.009306 25481 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.005034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.2.182.62:34761)
0924 10:48:58.005154 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.005158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.007926 (+ 2768us) server_negotiation.cc:407] Connection header received
0924 10:48:58.008317 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.008320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.008365 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.008410 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.009177 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46088: BlockingRecv error: recv got EOF from 127.0.0.1:46088 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:48:58.010159 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.005710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.2.182.61:43701)
0924 10:48:58.005822 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.005827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.008102 (+ 2275us) server_negotiation.cc:407] Connection header received
0924 10:48:58.008541 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.008544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.008587 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.008631 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.010060 (+ 1429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36520: BlockingRecv error: recv got EOF from 127.0.0.1:36520 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:48:58.014767 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.013278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.2.182.61:43701)
0924 10:48:58.013374 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.013378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.013867 (+ 489us) server_negotiation.cc:407] Connection header received
0924 10:48:58.014103 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.014105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.014170 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.014215 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.014671 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36524: BlockingRecv error: recv got EOF from 127.0.0.1:36524 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 10:48:58.019484 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.018188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.2.182.61:43701)
0924 10:48:58.018268 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.018272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.018683 (+ 411us) server_negotiation.cc:407] Connection header received
0924 10:48:58.018856 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.018858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.018899 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.018942 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.019378 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36538: BlockingRecv error: recv got EOF from 127.0.0.1:36538 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 10:48:58.024878 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.023526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.2.182.61:43701)
0924 10:48:58.023614 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.023618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.024088 (+ 470us) server_negotiation.cc:407] Connection header received
0924 10:48:58.024265 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.024269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.024310 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.024355 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.024800 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36546: BlockingRecv error: recv got EOF from 127.0.0.1:36546 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 10:48:58.032672 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.031310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.2.182.61:43701)
0924 10:48:58.031424 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.031428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.031889 (+ 461us) server_negotiation.cc:407] Connection header received
0924 10:48:58.032059 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.032062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.032104 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.032150 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.032592 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36552: BlockingRecv error: recv got EOF from 127.0.0.1:36552 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:48:58.044531 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.043019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.2.182.61:43701)
0924 10:48:58.043116 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.043120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.043656 (+ 536us) server_negotiation.cc:407] Connection header received
0924 10:48:58.043835 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.043838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.043881 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.043926 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.044434 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36566: BlockingRecv error: recv got EOF from 127.0.0.1:36566 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 10:48:58.070647 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.069345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.2.182.61:43701)
0924 10:48:58.069472 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.069477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.069695 (+ 218us) server_negotiation.cc:407] Connection header received
0924 10:48:58.069839 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.069842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.069902 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.069997 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.070524 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36568: BlockingRecv error: recv got EOF from 127.0.0.1:36568 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 10:48:58.105835 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.104629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.2.182.61:43701)
0924 10:48:58.104792 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.104797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.104970 (+ 173us) server_negotiation.cc:407] Connection header received
0924 10:48:58.105134 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.105138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.105192 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.105283 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.105725 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36578: BlockingRecv error: recv got EOF from 127.0.0.1:36578 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 10:48:58.173158 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.171943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.2.182.61:43701)
0924 10:48:58.172098 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.172104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.172248 (+ 144us) server_negotiation.cc:407] Connection header received
0924 10:48:58.172423 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.172427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.172484 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.172571 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.173045 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36582: BlockingRecv error: recv got EOF from 127.0.0.1:36582 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 10:48:58.304785 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.303580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.2.182.61:43701)
0924 10:48:58.303741 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.303747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.303882 (+ 135us) server_negotiation.cc:407] Connection header received
0924 10:48:58.304057 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.304060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.304116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.304216 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.304682 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36584: BlockingRecv error: recv got EOF from 127.0.0.1:36584 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 10:48:58.564232 25557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:48:58.563051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.2.182.61:43701)
0924 10:48:58.563174 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:58.563177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:58.563263 (+ 86us) server_negotiation.cc:407] Connection header received
0924 10:48:58.563387 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:58.563391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:58.563439 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:58.563605 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:48:58.564116 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36600: BlockingRecv error: recv got EOF from 127.0.0.1:36600 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:58.851351 25645 catalog_manager.cc:1259] Loaded cluster ID: 92cf71df70294679b7c946ccbd4e14ce
I20260924 10:48:58.851442 25645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: loading cluster ID for follower catalog manager: success
I20260924 10:48:58.853193 25645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: acquiring CA information for follower catalog manager: success
I20260924 10:48:58.854341 25645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d2ce8f151d64cf48ec8f15c8742973b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:48:58.871718 25660 catalog_manager.cc:1259] Loaded cluster ID: 92cf71df70294679b7c946ccbd4e14ce
I20260924 10:48:58.871819 25660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: loading cluster ID for follower catalog manager: success
I20260924 10:48:58.873622 25660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: acquiring CA information for follower catalog manager: success
I20260924 10:48:58.874809 25660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a53ba7d8de2d4a76a3721793dda0b7bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:48:59.082121 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:59.078337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.2.182.61:43701)
0924 10:48:59.078601 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:59.078608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:59.078628 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:48:59.078812 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:59.078814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:59.078938 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:59.079049 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:59.079296 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.079821 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.080653 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.080880 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.081021 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:59.081311 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:59.081732 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:59.081879 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:59.081913 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 10:48:59.081965 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260924 10:48:59.088603 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:59.084972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.2.182.60:46805)
0924 10:48:59.085249 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:59.085254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:59.085275 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:48:59.085440 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:59.085442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:59.085552 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:59.085674 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:59.085875 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.086361 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.087189 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.087434 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.087565 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:59.087845 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:59.088211 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:59.088335 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:59.088397 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:48:59.088455 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260924 10:48:59.094892 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:59.091524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.2.182.61:43701)
0924 10:48:59.091631 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:59.091640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:59.091653 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:48:59.091795 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:59.091798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:59.091885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:59.091969 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:59.092215 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.092988 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.093905 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.094122 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.094219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:59.094445 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:59.094502 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:59.094573 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:59.094677 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 10:48:59.094720 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:48:59.095638 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:59.091278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.2.182.60:46805)
0924 10:48:59.091379 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:59.091384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:59.091909 (+ 525us) server_negotiation.cc:407] Connection header received
0924 10:48:59.092058 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:59.092061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:59.092108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:59.092167 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:59.092534 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.093066 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.094276 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.094598 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.094671 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:59.095172 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:59.095239 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:59.095283 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:59.095459 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 10:48:59.095505 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:48:59.095952 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:48:59.091168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.2.182.62:34761)
0924 10:48:59.091375 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:48:59.091381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:48:59.091603 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:48:59.091909 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:48:59.091912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:48:59.092056 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:48:59.092174 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:48:59.092439 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.093209 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.094276 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:48:59.094601 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:48:59.094684 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:48:59.095210 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:48:59.095505 (+ 295us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:48:59.095580 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:48:59.095750 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:48:59.095817 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":54,"threads_started":1}
I20260924 10:48:59.098767 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25409
I20260924 10:48:59.104883 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25484
I20260924 10:48:59.110332 2776 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwg6lLW/build/debug/bin/kudu with pid 25558
2026-09-24T10:48:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3026 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6017 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276508 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST